[recovery]: channels in Limbo after restore

[warioishere]

Background

After our Server crashed and the datapool got corrupted, we stored our btcpay-server from a two day told backup. After the node was beeing startet, our channels were gone and been put into "waiting close" according to rtl

All channel funds from my side are stuck in Limbo.

The state after two ours is still "waiting close"

Your environment

• version of lnd
  lnd 17.2beta

• which operating system (uname -a on *Nix)
  lxc ubuntu 22.04.3 on Proxmox kernel 6.5.0-14-generic

• version of btcd, bitcoind, or other backend
  26.0

• any other relevant environment details
  btcpay-server docker latest (1.17)

root@btcpay:~/btcpayserver-docker# bitcoin-lncli.sh pendingchannels

{
    "total_limbo_balance":  "7631509",
    "pending_open_channels":  [],
    "pending_closing_channels":  [],
    "pending_force_closing_channels":  [],
    "waiting_close_channels":  [
        {
            "channel":  {
                "remote_node_pub":  "0241eae1ac5fd3f08567cf4ca579b076cc96d3c4f35c138379a99a25b2c26e7a70",
                "channel_point":  "b0b637a872daa0063a4a1232c6d7a0ce504e96da8e5391c38c83bd5641852aa6:0",
                "capacity":  "262102",
                "local_balance":  "56153",
                "remote_balance":  "198845",
                "local_chan_reserve_sat":  "2621",
                "remote_chan_reserve_sat":  "2621",
                "initiator":  "INITIATOR_REMOTE",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "0",
                "chan_status_flags":  "ChanStatusLocalDataLoss",
                "private":  true,
                "memo":  ""
            },
            "limbo_balance":  "56153",
            "commitments":  {
                "local_txid":  "6b35d3fc944eeeec157646a005d18418adc1ecfc1cd2340daf83f832c99ce5b9",
                "remote_txid":  "6e38f9fca3e73fccf67351e37a65e65ffedd60f7f5488392e793172e0dbdc71f",
                "remote_pending_txid":  "",
                "local_commit_fee_sat":  "6443",
                "remote_commit_fee_sat":  "6443",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  ""
        },
        {
            "channel":  {
                "remote_node_pub":  "0241eae1ac5fd3f08567cf4ca579b076cc96d3c4f35c138379a99a25b2c26e7a70",
                "channel_point":  "d1f7cddc18357be33fb26567c43e6cf56b41f63e39d57bebf3614ac512b67ee6:0",
                "capacity":  "500000",
                "local_balance":  "496530",
                "remote_balance":  "0",
                "local_chan_reserve_sat":  "5000",
                "remote_chan_reserve_sat":  "5000",
                "initiator":  "INITIATOR_LOCAL",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "0",
                "chan_status_flags":  "ChanStatusLocalDataLoss",
                "private":  false,
                "memo":  ""
            },
            "limbo_balance":  "496530",
            "commitments":  {
                "local_txid":  "3e1bfcf72032c46d0ec6654fbe6fd64bb48c5271751160dbda383a91e96f39a2",
                "remote_txid":  "5df4540b40d03f983e341ebd66536ac8c138ad7ff5e8ee0fd2213cf0fb9fb28f",
                "remote_pending_txid":  "",
                "local_commit_fee_sat":  "2810",
                "remote_commit_fee_sat":  "2810",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  ""
        },
        {
            "channel":  {
                "remote_node_pub":  "027222caea9c1070952bc8c7a05a6f75f26f3bffc37e80e627cc651d50d50eb268",
                "channel_point":  "9110b04f1ec66f20e3d7a8e0a8e15ce10f0a4c8729872c3a5cd036093829ef4b:1",
                "capacity":  "2000000",
                "local_balance":  "1996530",
                "remote_balance":  "0",
                "local_chan_reserve_sat":  "20000",
                "remote_chan_reserve_sat":  "20000",
                "initiator":  "INITIATOR_LOCAL",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "0",
                "chan_status_flags":  "ChanStatusLocalDataLoss",
                "private":  false,
                "memo":  ""
            },
            "limbo_balance":  "1996530",
            "commitments":  {
                "local_txid":  "4c0123a75a6171302bacf172cd67cfbd0556ef31626a380ca92c31adc0807b4a",
                "remote_txid":  "eeed128e5d8a095ee53a52e56e734b5a4547ae51c34b93aac80ae2d36e75b1b2",
                "remote_pending_txid":  "",
                "local_commit_fee_sat":  "2810",
                "remote_commit_fee_sat":  "2810",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  ""
        },
        {
            "channel":  {
                "remote_node_pub":  "02765a281bd188e80a89e6ea5092dcb8ebaaa5c5da341e64327e3fadbadcbc686c",
                "channel_point":  "7cb8fbb2aafc9d2c3e7badf2e68b39fff9c48d65ed059057aee924a160cd88aa:0",
                "capacity":  "6000000",
                "local_balance":  "120269",
                "remote_balance":  "5872600",
                "local_chan_reserve_sat":  "60000",
                "remote_chan_reserve_sat":  "60000",
                "initiator":  "INITIATOR_REMOTE",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "0",
                "chan_status_flags":  "ChanStatusLocalDataLoss",
                "private":  false,
                "memo":  ""
            },
            "limbo_balance":  "120269",
            "commitments":  {
                "local_txid":  "84bbd791cb7dbb3655c0eaff3bd67bc678b8e40a21abac5e5863df38b6d4f295",
                "remote_txid":  "ffe561d1c860741ef43e323d52007d9bc47c92b1f7bf0e36208e6170bd70dc8b",
                "remote_pending_txid":  "",
                "local_commit_fee_sat":  "6470",
                "remote_commit_fee_sat":  "6470",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  ""
        },
        {
            "channel":  {
                "remote_node_pub":  "02765a281bd188e80a89e6ea5092dcb8ebaaa5c5da341e64327e3fadbadcbc686c",
                "channel_point":  "5146db63a8beaf8c590d72a4aee3485bec1b726d98484cfdc057ff013d60b0f1:1",
                "capacity":  "5000000",
                "local_balance":  "4962027",
                "remote_balance":  "34502",
                "local_chan_reserve_sat":  "50000",
                "remote_chan_reserve_sat":  "50000",
                "initiator":  "INITIATOR_LOCAL",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "0",
                "chan_status_flags":  "ChanStatusLocalDataLoss",
                "private":  false,
                "memo":  ""
            },
            "limbo_balance":  "4962027",
            "commitments":  {
                "local_txid":  "26a3e9239f4feac30868ccd630b9fcf86dd2cda7fae43dc19ecbefd63b9404d0",
                "remote_txid":  "6a5e855baaedc6710996c4d43bce590499923398661cb7758381519ca1f9c29f",
                "remote_pending_txid":  "",
                "local_commit_fee_sat":  "2810",
                "remote_commit_fee_sat":  "2810",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  ""
        }
    ]
}

part of lnd logs:

2023-12-16 10:42:01.082 [ERR] LNWL: ChannelPoint(7cb8fbb2aafc9d2c3e7badf2e68b39fff9c48d65ed059057aee924a160cd88aa:0): sync failed with local data loss: remote believes our tail height is 1728, while we have 1710!
2023-12-16 10:42:01.082 [WRN] HSWC: ChannelLink(7cb8fbb2aafc9d2c3e7badf2e68b39fff9c48d65ed059057aee924a160cd88aa:0): error when syncing channel states: ChannelPoint(7cb8fbb2aafc9d2c3e7badf2e68b39fff9c48d65ed059057aee924a160cd88aa:0) with CommitPoint(03994b7b04841f4f4e6d5fa09aa73738a5e1a9ea529ecd1e6db3b1954eba894462) had possible local commitment state data loss
2023-12-16 10:42:01.082 [INF] HSWC: ChannelLink(5146db63a8beaf8c590d72a4aee3485bec1b726d98484cfdc057ff013d60b0f1:1): received re-establishment message from remote side
2023-12-16 10:42:01.082 [ERR] LNWL: ChannelPoint(5146db63a8beaf8c590d72a4aee3485bec1b726d98484cfdc057ff013d60b0f1:1): sync failed with local data loss: remote believes our tail height is 55, while we have 52!
2023-12-16 10:42:01.082 [WRN] HSWC: ChannelLink(5146db63a8beaf8c590d72a4aee3485bec1b726d98484cfdc057ff013d60b0f1:1): error when syncing channel states: ChannelPoint(5146db63a8beaf8c590d72a4aee3485bec1b726d98484cfdc057ff013d60b0f1:1) with CommitPoint(0396a00e366fbc9a9eda3f39506b2a3df05d5e7ac4f532a1776d0ac6aac3442292) had possible local commitment state data loss
2023-12-16 10:42:01.083 [ERR] HSWC: ChannelLink(7cb8fbb2aafc9d2c3e7badf2e68b39fff9c48d65ed059057aee924a160cd88aa:0): failing link: unable to synchronize channel states: ChannelPoint(7cb8fbb2aafc9d2c3e7badf2e68b39fff9c48d65ed059057aee924a160cd88aa:0) with CommitPoint(03994b7b04841f4f4e6d5fa09aa73738a5e1a9ea529ecd1e6db3b1954eba894462) had possible local commitment state data loss with error: unable to resume channel, recovery required
2023-12-16 10:42:01.083 [INF] HSWC: ChannelLink(7cb8fbb2aafc9d2c3e7badf2e68b39fff9c48d65ed059057aee924a160cd88aa:0): exited
2023-12-16 10:42:01.083 [INF] HSWC: ChannelLink(7cb8fbb2aafc9d2c3e7badf2e68b39fff9c48d65ed059057aee924a160cd88aa:0): stopping
2023-12-16 10:42:01.083 [INF] HSWC: Removing channel link with ChannelID(aa88cd60a124e9ae579005ed658dc4f9ff398be6f2ad7b3e2c9dfcaab2fbb87c)
2023-12-16 10:42:01.092 [ERR] HSWC: ChannelLink(5146db63a8beaf8c590d72a4aee3485bec1b726d98484cfdc057ff013d60b0f1:1): failing link: unable to synchronize channel states: ChannelPoint(5146db63a8beaf8c590d72a4aee3485bec1b726d98484cfdc057ff013d60b0f1:1) with CommitPoint(0396a00e366fbc9a9eda3f39506b2a3df05d5e7ac4f532a1776d0ac6aac3442292) had possible local commitment state data loss with error: unable to resume channel, recovery required

can anyone help me to recover the funds because im quiet nervous as we have quiet some sats in those channels..
its just annoying if a backup strategy doesnt work and i didnt think a two day old backup would kill all my channels. I recovered already a one day old backup somewhen else and nothing happend, everything worked as expected.

Goal
I dont want to forceclose if possible. Fees are two high and our channel partners are healthy and we would like to stay in touch.

Additional Infos, as far as i know, the channel balances are still the same today as the two day old backup.
Last channelbackup from rtl is from dez04 with all one channel missing from the current state

Can anyone suggest what to do?

[guggero]

Restoring from a file based backup is BAD, very BAD, basically the worst thing you can do in this situation: https://github.com/lightningnetwork/lnd/blob/master/docs/safety.md#file-based-backups

Do not under any circumstances force close any channels, lnd not doing that right now is the only reason you haven't lost all the money in the channels. If you force close now, your channels will most likely be breached and the other party can take all funds because of the outdated states.

Shut down that lnd and then follow the SCB recovery procedure described here: https://github.com/lightningnetwork/lnd/blob/master/docs/recovery.md
This will ask the remote peers to force close (the channels will not become operational again at this point), so fees will be ugly... I guess you can wait with initiating the SCB recovery until fees go down somewhat. But make sure you stop the current lnd that uses the outdated database from the file based backup.

[warioishere]

@guggero

Damn it, i didnt know this can go wrong, as restoring from an backup that was an day old never made problems :( i didnt know about this. I am stupid and not well prepared I see :(

I got an replication from HA enviroment this morning, destroyed this old container and created a new container with a replication from arround 01.30 this morning. Is if any use if i go online with that state?

If no i will try to do this SCB thing?

[warioishere]

@guggero

I just want to be sure, on my exit server, all i need to do is

lncli restorechanbackup --multi_backup=channel.backup

just wait for the right time because of the fees? Will i be asked for a fee rate or is it automatic? I guess i pay those fees completly as its my fault in that situation?

maybe you could say a thing if i could try coming online with a more recent backup from this morning (01.30AM) its 15.00 here now.

[guggero]

Is if any use if i go online with that state?

I wouldn't risk it. At this point the peers very likely started to force close the channels already as you came up with an outdated channel state.

I just want to be sure, on my exit server, all i need to do is

You need to do this on a fresh lnd install, that is restored from seed, so with a completely empty data directory.

maybe you could say a thing if i could try coming online with a more recent backup from this morning (01.30AM) its 15.00 here now.

No. A file based backup should NEVER be used, no matter how recent it is. In the worst case a couple of milliseconds are enough to make it outdated and therefore toxic.

[warioishere]

@guggero

i dont want to argue or question what you say, but why do I have to do this from a fresh install?
Im running btcpay docker which makes things a bit more complicated as new wallets are automatically created during install. Im not very much into all those things so im pretty worried now... whats the problem running SCB recovery from the current server with outdate channel states where I stopped LND now?

can I also just remove the lnd docker container, install the LND docker again and the replace files in

/data/data/chain/bitcoin/mainnet/ ?

if yes, all files?

to make things clear, im worried to create a new btcpay server install and replace those files, I dont want to fuck up things more as they are. I just wanna be sure everything goes well.

PS: ich bin us de schwiiz, vielleicht magsch mir persönlich helfe, bin es bitz nervös :(
0799351192 (whatsapp/telegram/signal)

[guggero]

You can use the same installation, but you need to remove the channel.db file first to make sure you don't accidentally publish one of those outdated states. Once you removed the channel.db file from the btcpay server installation, there will be zero channels being shown (which is to be expected) but your on-chain balance should still be there. Then you apply the SCB file (lncli restorechanbackup) and observe them in lncli pendingchannels.

Sorry, had a long week, need the down time during the weekend. It's all documented fairly well I think, so you should be able to do this. If not, I'll be around on Monday.

[warioishere]

@guggero

just remove channel.db and then use the recovery commands?

what about the rest:

803dcb2a92e0:/data/data/chain/bitcoin/mainnet# ls -l
total 657

-rw-r--r--    1 root     root            83 Nov 24  2022 chainnotifier.macaroon
-rw-r--r--    1 root     root          1949 Dec 14 22:56 channel.backup
-rw-r--r--    1 root     root            91 Nov 24  2022 invoices.macaroon
-rw-------    1 root     root         20480 Dec 14 22:56 macaroons.db
-rw-r--r--    1 root     root            91 Nov 24  2022 router.macaroon
-rw-r--r--    1 root     root            92 Nov 24  2022 signer.macaroon
-rw-------    1 root     root       1892352 Dec 16 00:29 wallet.db
-rw-r--r--    1 root     root           114 Nov 24  2022 walletkit.macaroon
-rw-r--r--    1 root     root           261 Nov 24  2022 walletunlock.json

sorry i know you have deserved your weekend.

Just let me know if you have time to verify:

just delete channal.db

use lncli restorechanbackup

does it matter which files of channel.back i use? i guess no? i have this one from today, and the one from two days ago

viele viele dank für dini müe!!

[guggero]

Yes, stop lnd, delete data/graph/mainnet/channel.db, leave the rest of the files. Then start lnd and lncli restorechanbackup.
No it doesn't matter what channel.backup you use, but the most recent is usually the best.

[warioishere]

PS no im reading the docs but its not so clear what i have to do in case of such accidents. I didnt do this on purpose. And also i didnt know if i can do this on the exisiting server, or on a fresh install.

[warioishere]

Yes, stop lnd, delete data/graph/mainnet/channel.db, leave the rest of the files. Then start lnd and lncli restorechanbackup. No it doesn't matter what channel.backup you use, but the most recent is usually the best.

okay thanks, ill set the command when the fees are lower. Ill let you know if it worked out.
Schöööhs weekend wünsch i no us Bülach

[warioishere]

@guggero
Hi Oliver. I wanted to check the channels today and initiate SCB because fees came back already a bit. Sadly, two of the peers already initiated a force close of the channels where I also have SATs in.
Can I continue deleting channel.db and initiate recovery with the multi channel file or should I just initiate SCB on the channels which are still pending waiting to close and leave those which are already force-closing?
I mean if I delete channel dB, those that are force closing are also gone?
I'm unsure what to do :(
Greetings

[warioishere]

got it to work, delete channel.db, restartet the container, all channels are gone
did sucessfully the scb restore but now:

root@btcpay:~/btcpayserver-docker# ./bitcoin-lncli.sh pendingchannels

{
    "total_limbo_balance":  "0",
    "pending_open_channels":  [],
    "pending_closing_channels":  [],
    "pending_force_closing_channels":  [],
    "waiting_close_channels":  [
        {
            "channel":  {
                "remote_node_pub":  "0241eae1ac5fd3f08567cf4ca579b076cc96d3c4f35c138379a99a25b2c26e7a70",
                "channel_point":  "d1f7cddc18357be33fb26567c43e6cf56b41f63e39d57bebf3614ac512b67ee6:0",
                "capacity":  "500000",
                "local_balance":  "0",
                "remote_balance":  "0",
                "local_chan_reserve_sat":  "0",
                "remote_chan_reserve_sat":  "0",
                "initiator":  "INITIATOR_LOCAL",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "0",
                "chan_status_flags":  "ChanStatusLocalDataLoss|ChanStatusRestored",
                "private":  true,
                "memo":  ""
            },
            "limbo_balance":  "0",
            "commitments":  {
                "local_txid":  "",
                "remote_txid":  "",
                "remote_pending_txid":  "",
                "local_commit_fee_sat":  "0",
                "remote_commit_fee_sat":  "0",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  ""
        },
        {
            "channel":  {
                "remote_node_pub":  "027222caea9c1070952bc8c7a05a6f75f26f3bffc37e80e627cc651d50d50eb268",
                "channel_point":  "9110b04f1ec66f20e3d7a8e0a8e15ce10f0a4c8729872c3a5cd036093829ef4b:1",
                "capacity":  "2000000",
                "local_balance":  "0",
                "remote_balance":  "0",
                "local_chan_reserve_sat":  "0",
                "remote_chan_reserve_sat":  "0",
                "initiator":  "INITIATOR_LOCAL",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "0",
                "chan_status_flags":  "ChanStatusLocalDataLoss|ChanStatusRestored",
                "private":  true,
                "memo":  ""
            },
            "limbo_balance":  "0",
            "commitments":  {
                "local_txid":  "",
                "remote_txid":  "",
                "remote_pending_txid":  "",
                "local_commit_fee_sat":  "0",
                "remote_commit_fee_sat":  "0",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  ""
        },
        {
            "channel":  {
                "remote_node_pub":  "02765a281bd188e80a89e6ea5092dcb8ebaaa5c5da341e64327e3fadbadcbc686c",
                "channel_point":  "5146db63a8beaf8c590d72a4aee3485bec1b726d98484cfdc057ff013d60b0f1:1",
                "capacity":  "5000000",
                "local_balance":  "0",
                "remote_balance":  "0",
                "local_chan_reserve_sat":  "0",
                "remote_chan_reserve_sat":  "0",
                "initiator":  "INITIATOR_LOCAL",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "0",
                "chan_status_flags":  "ChanStatusLocalDataLoss|ChanStatusRestored",
                "private":  true,
                "memo":  ""
            },
            "limbo_balance":  "0",
            "commitments":  {
                "local_txid":  "",
                "remote_txid":  "",
                "remote_pending_txid":  "",
                "local_commit_fee_sat":  "0",
                "remote_commit_fee_sat":  "0",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  ""
        }
    ]
}

[warioishere]

ich nimm mal ah, es isch alles weg

rtl zeigt auch nix mehr an :(

[guggero]

The channels now need to close on chain, which can take quite a while in the current fee market. But no sats should be lost if the channel peers are online and responding.

[warioishere]

@guggero
shouldnt there be a transactionID seen? its empty. and my lnd was full of error logs
these logs are endless long and they did read as things work out properly:
its one of the pending channels (d1f7cddc18357be33fb26567c43e6cf56b41f63e39d57bebf3614ac512b67ee6:0)

2023-12-18 22:11:59.022 [INF] PEER: Peer(0241eae1ac5fd3f08567cf4ca579b076cc96d3c4f35c138379a99a25b2c26e7a70): Loading ChannelPoint(d1f7cddc18357be33fb26567c43e6cf56b41f63e39d57bebf3614ac512b67ee6:0), isPending=false
2023-12-18 22:11:59.022 [WRN] PEER: Peer(0241eae1ac5fd3f08567cf4ca579b076cc96d3c4f35c138379a99a25b2c26e7a70): Unable to find our forwarding policy for channel d1f7cddc18357be33fb26567c43e6cf56b41f63e39d57bebf3614ac512b67ee6:0, using default values
2023-12-18 22:11:59.022 [INF] HSWC: ChannelLink(d1f7cddc18357be33fb26567c43e6cf56b41f63e39d57bebf3614ac512b67ee6:0): starting
2023-12-18 22:11:59.022 [INF] HSWC: Trimming open circuits for chan_id=820415:2938:0, start_htlc_id=0
2023-12-18 22:11:59.022 [INF] HSWC: Adding live link chan_id=e67eb612c54a61f3eb7bd5393ef6416bf56c3ec46765b23fe37b3518dccdf7d1, short_chan_id=820415:2938:0
2023-12-18 22:11:59.022 [INF] CNCT: Attempting to update ContractSignals for ChannelPoint(d1f7cddc18357be33fb26567c43e6cf56b41f63e39d57bebf3614ac512b67ee6:0)
2023-12-18 22:11:59.022 [INF] HSWC: ChannelLink(d1f7cddc18357be33fb26567c43e6cf56b41f63e39d57bebf3614ac512b67ee6:0): HTLC manager started, bandwidth=0 mSAT
2023-12-18 22:11:59.022 [INF] HSWC: ChannelLink(d1f7cddc18357be33fb26567c43e6cf56b41f63e39d57bebf3614ac512b67ee6:0): attempting to re-synchronize
2023-12-18 22:11:59.022 [INF] PEER: Peer(0241eae1ac5fd3f08567cf4ca579b076cc96d3c4f35c138379a99a25b2c26e7a70): Negotiated chan series queries
2023-12-18 22:11:59.022 [INF] DISC: Creating new GossipSyncer for peer=0241eae1ac5fd3f08567cf4ca579b076cc96d3c4f35c138379a99a25b2c26e7a70
2023-12-18 22:11:59.022 [INF] NTFN: New block epoch subscription
2023-12-18 22:11:59.070 [INF] HSWC: ChannelLink(d1f7cddc18357be33fb26567c43e6cf56b41f63e39d57bebf3614ac512b67ee6:0): received re-establishment message from remote side
2023-12-18 22:11:59.070 [ERR] LNWL: **ChannelPoint(d1f7cddc18357be33fb26567c43e6cf56b41f63e39d57bebf3614ac512b67ee6:0): sync failed with local data loss: remote believes our tail height is 14, while we have 0!**
2023-12-18 22:11:59.070 [WRN] LNWL: ChannelPoint(d1f7cddc18357be33fb26567c43e6cf56b41f63e39d57bebf3614ac512b67ee6:0): detected restored triggering DLP
2023-12-18 22:11:59.070 [WRN] HSWC: ChannelLink(d1f7cddc18357be33fb26567c43e6cf56b41f63e39d57bebf3614ac512b67ee6:0): error when syncing channel states: **ChannelPoint(d1f7cddc18357be33fb26567c43e6cf56b41f63e39d57bebf3614ac512b67ee6:0) with CommitPoint(02f16b37c9a367a588349637142037c91a2c4dd82f0e47995de756b00f5d57adcf) had possible local commitment state data loss**
2023-12-18 22:11:59.071 [ERR] HSWC: ChannelLink(d1f7cddc18357be33fb26567c43e6cf56b41f63e39d57bebf3614ac512b67ee6:0): failing link: unable to synchronize channel states: ChannelPoint(d1f7cddc18357be33fb26567c43e6cf56b41f63e39d57bebf3614ac512b67ee6:0) with CommitPoint(02f16b37c9a367a588349637142037c91a2c4dd82f0e47995de756b00f5d57adcf) had possible local commitment state data loss with error: unable to resume channel, recovery required
2023-12-18 22:11:59.071 [INF] HSWC: ChannelLink(d1f7cddc18357be33fb26567c43e6cf56b41f63e39d57bebf3614ac512b67ee6:0): exited
2023-12-18 22:11:59.071 [INF] HSWC: ChannelLink(d1f7cddc18357be33fb26567c43e6cf56b41f63e39d57bebf3614ac512b67ee6:0): stopping
2023-12-18 22:11:59.071 [INF] HSWC: Removing channel link with ChannelID(e67eb612c54a61f3eb7bd5393ef6416bf56c3ec46765b23fe37b3518dccdf7d1)
2023-12-18 22:12:06.215 [ERR] DISC: Unable to query bootstrapper BOLT-0010 DNS Seed: [[nodes.lightning.directory soa.nodes.lightning.directory] [lseed.bitcoinstats.com ]]: tor host is unreachable

[warioishere]

[guggero]

There are transactions:
https://mempool.space/tx/72afc30a1f0ac97dda6b48536160eb9971f479555344dbbd8e62e571036d003d
https://mempool.space/tx/ec211576e526c6f44509ede7d96ca319c49afb60f2bbc30c3f78e55c350d87e3

But your node won't see or react to them until they confirm (because of the recovery mode).
You can find these transactions by taking the TXID from the channel point and enter them in mempool.space.

The errors you see in the log are to be expected, sync failed with local data loss: and detected restored triggering DLP means your peer is reacting to you using the SCB and is initiating the force close, which is what you want.

[warioishere]

okay thanks Oliver, I see. That might take several days, if not for ever. I hope fees come down that much again.
Pls send me your lightning adress or send me a link of an invoice of 25k sats so that I can thank you for your help.

Another question, is it save to open new channels while in recovery Node. We have an onlineshop and we cant accept any Lightning Payments currently :(

[guggero]

You're welcome. No need to send any sats, but thank you anyway!

Yes, it's safe to open new channels. Just make sure your file system is secured so a file-based backup isn't necessary in the future (making sure the channel.db file is never corrupted is the most important step for any node with funds in channels). And please do read the full safety doc at some point!

[warioishere]

@guggero is there any way to bump fees for such a transaction? I am certainly afraid Ill never get this back

[guggero]

You can use the new chantools pullanchor command from this PR: lightninglabs/chantools#100
You'll need to build chantools from source though, but you can run it on any computer you want (you'll need access to the lnd seed). Also make sure to read the example in the pull request description on how to use the command.

[warioishere]

thank you @guggero

i suceeded to install the pullanchor PR

but im still afraid to use this command as I am not 100%sure what those
means

chantools pullanchor \
    --sponsorinput <txid:vout> \        # must be an input from lncli listunspent that is confirmed and has enough funds to pay for fees
    --anchoraddr bc1q... \        # this is the on-chain address of the anchor output (the one with 330 satoshis)
    --changeaddr bc1q... \        # this is a p2wkh address from your lnd wallet where the change goes to (lncli newaddress p2wkh)
    --feerate xx        # the desired fee rate in sat/vByte, must be high enough to sponsor both transactions, so must be higher than the "effective" fee rate currently is

Am I right with the follwing:
run chantools pullanchor
type in seedphrase

the command asks me fore the --values?
or do i have to make the command like

chantools pullanchor --sponsorinput=xxx --anchoraddr=xxx --changeaddr=xxx --feerate=xxx

Example Transaction one:

https://mempool.space/tx/72afc30a1f0ac97dda6b48536160eb9971f479555344dbbd8e62e571036d003d

sponsor input --> cd683b142301b8a2fbe2edf852d9d43a0b35c0f1a565bab92cb8c6a88d0e0ff5:0 (utxo from my lndwallet)
anchor adresse --> bc1qnm00fwzckjnlqj6nghpegatx492ttkglxahhehyemm3aaw7cqmzshls39v
change adresse --> any adress of my lightning wallet
feerate --> for example 100

Example Transaction two:

same but what is the anchor addresse? it has two?

https://mempool.space/tx/ec211576e526c6f44509ede7d96ca319c49afb60f2bbc30c3f78e55c350d87e3

Example 3

this channel:

{
                "remote_node_pub":  "0241eae1ac5fd3f08567cf4ca579b076cc96d3c4f35c138379a99a25b2c26e7a70",
                "channel_point":  "d1f7cddc18357be33fb26567c43e6cf56b41f63e39d57bebf3614ac512b67ee6:0",
                "capacity":  "500000",
                "local_balance":  "0",
                "remote_balance":  "0",
                "local_chan_reserve_sat":  "0",
                "remote_chan_reserve_sat":  "0",
                "initiator":  "INITIATOR_LOCAL",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "0",
                "chan_status_flags":  "ChanStatusLocalDataLoss|ChanStatusRestored",
                "private":  true,
                "memo":  ""
            },

I didnt find the transaction ID of the force close yet, how did you find those? can those be extracted from where?
If i typ in the channelpoint ID into mempool then the opening transactions appears, and in that transaction i cant find anything that refers to the close.

Sorry for those many questions, i really dont want to mess anything up :(

thanks and happy, calming and relaxing christmas days Oliver!

[warioishere]

as far as i understand thist:

I couldn't find a way to tell which anchor output is the local one, is there an easy way to see this, or does one just have to try?

Unfortunately there is no way to tell AFAIK. But since we try to derive the address first, it will fail if no match can be found, then you know to use the other one.

so Ill just give it a try, maybe you can help me how to use the command properly because I have trouble to understand the syntax and how to find the txID of the last channel

[warioishere]

i forced myself to just do it, and i know how the tool works now, but im asking myself why i should use a utxo that is higher then 84671sats?
2023-12-25 10:14:27.071 [INF] CHAN: Fee 16900 sats of 84671 total amount (estimated weight 676)
i set a fee of 100 and thats quiet alot of money?
i used a utxo of 80k sats and it was rejected.

[warioishere]

update:

ive found out why channel number two ist not im mempool, so I wrote the owner of it directly if they could manually close it.

theres one last question left. When I enter feerate of lets say 100sats/vbyte, chantool sents me:

2023-12-25 10:14:27.071 [INF] CHAN: Fee 16900 sats of 84671 total amount (estimated weight 676)

so thats arround 85k sats. (100x676 plus 16900 ~84500sats. Why is the size of this transaction so high? is it possible that i need to spend 30Dollars für that closing??

I used a utxo with arround 84k sats and the transaction was refused

[warioishere]

update2:

i dont care about the fees anymore, I am using a utxo with enough sats on it, but i always get:

[lncli] rpc error: code = Unknown desc = transaction rejected: output already spent

ich verzweifle hier langsam bei der Stange Geld die da festsitzt

example utxo:

11752eca409872544f218bb349292c3647baf7955877b89d29931a5455c5d650:0

no matter how many confirmations it has and what utxo i use from the ln wallet (lncli listunspent), it says always ouput already spent :(

[warioishere]

@guggero Hi Oliver, i hope you had a nice christmas time, maybe you dont see my messages thatswhy I am trying again tagging you. I dont want to bother you but i need our funds back for our store.

Current situation as in the last post:

[lncli] rpc error: code = Unknown desc = transaction rejected: output already spent

example utxo

11752eca409872544f218bb349292c3647baf7955877b89d29931a5455c5d650:0

not matter what utxo I use, i get an output already spent

Again I dont want to bother you with asking and asking but its just a big pain for me and my startup

[guggero]

Is your wallet fully synced? The UTXO you showed me has indeed already been used (you can check on a block explorer). Maybe you need to use --reset-wallet-transactions on your lnd node to tell it to re-scan the chain. Or you go through each of your UTXOs and find one that is indeed not spent yet (and do the re-scanning later when all channels were resolved).

[warioishere]

Hallo Oli, thanks for your answer. the utxo in the example above had been spend afterwards. I always used unspend utxos. I solved the problem in the maintime: The problem was, that the closing tx was not in my own mempool anymore due to highblockchain usegage in the last monts, I had to republish it and raise the maxmempool.

After that, the tx created with chantools and finalize psbt could be published. Thank you and I wish you a good day.
Greetings
Mario