How to resurrect zombie edges?

[hazrulnizam]

I have two lnd nodes NodeA and NodeB:

	NodeA	NodeB
lnd	v0.17.3-beta	v0.17.3-beta
backend	bitcoind	neutrino
database	boltdb	postgreSQL
channels	20+	0 for a long time. Now 1.

NodeA is my primary node where I give most of my attention to while NodeB is for trying out different configs.

NodeB used to have channels to multiple nodes but then I closed all of them and it had been running for more than a year now with no open channels.

Now I want to use NodeB for a project, so I opened a channel between NodeA and NodeB. My hope is that transactions can route to and from NodeB via NodeA.

Transactions going into NodeB works fine. The problem comes when trying to send out payments from NodeB.

Almost all of the open channels on NodeA are now marked as zombies in NodeB's graph (presumeably because it ran without open channels for so long), so payment attempts end up with NO_ROUTE errors:

[lnd@NodeB ~]$ lncli getchaninfo <chan1>
[lncli] rpc error: code = Unknown desc = edge marked as zombie
[lnd@NodeB ~]$ lncli getchaninfo <chan2>
[lncli] rpc error: code = Unknown desc = edge marked as zombie
[lnd@NodeB ~]$ lncli getchaninfo <chan3>
[lncli] rpc error: code = Unknown desc = edge marked as zombie
[lnd@NodeB ~]$ lncli getchaninfo <chan4>
[lncli] rpc error: code = Unknown desc = edge marked as zombie
[lnd@NodeB ~]$ lncli getchaninfo <chan5>
[lncli] rpc error: code = Unknown desc = edge marked as zombie

[lnd@NodeB ~]$ lncli payinvoice <payreq>
Payment hash: [REDACTED]
Description:
Amount (in satoshis): 10
Fee limit (in satoshis): 10
Destination: <some node connected to NodeA>
Confirm payment (yes/no): yes
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT | ROUTE |
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
Amount + fee:   0 + 0 sat
Payment hash:   [REDACTED]
Payment status: FAILED, reason: FAILURE_REASON_NO_ROUTE
[lncli] FAILED

I need to mark these edges as live on NodeB's graph.

I have tried:

1. Using gossip.pinned-syncers=NodeA in NodeB's conf.
2. Sending channel updates from NodeA, by updating the channels' policy.
3. Restarting NodeA
4. Restarting NodeB
5. Waiting. It has been more than 24 hours.

but the edges are still marked as zombies.

I know chantools can drop the graph but this is only for boltdb nodes. I need something similar but for postgreSQL nodes.

Do edges get marked live automatically? If they do, how long do I need to wait? Can I do something to expedite it?

Appreciate any ideas.

---

UPDATE: It has been 2 days since I posted the above (which I assume should be enough time for new channel updates to come in). However, the channels are still marked as zombies.

[bitromortac]

The channels should get removed from the zombie index after new channel updates are seen https://github.com/bitromortac/lnd/blob/master/discovery/gossiper.go#L2758, I'll try to reproduce your case.

Have you seen any log statements Removed edge with chan_id?

[hazrulnizam]

Unfortunately due to maxlogfiles left at default 3, the remaining logs no longer contain the period they would have been purged.

[lnd@NodeB ~]$ cd .lnd/logs/bitcoin/mainnet/
[lnd@NodeB mainnet]$ ls -larth
total 13M
drwx------. 3 lnd lnd   21 Nov 22  2021 ..
-rw-r--r--. 1 lnd lnd 1.3M Oct 25 23:57 lnd.log.20.gz
-rw-r--r--. 1 lnd lnd 1.3M Nov 20 14:52 lnd.log.21.gz
-rw-r--r--. 1 lnd lnd 1.3M Dec 25 19:40 lnd.log.22.gz
drwx------. 2 lnd lnd   84 Dec 25 19:40 .
-rw-r--r--. 1 lnd lnd 8.5M Jan 24 18:09 lnd.log
[lnd@NodeB mainnet]$ zcat lnd.log.20.gz | grep Removed
[lnd@NodeB mainnet]$ zcat lnd.log.21.gz | grep Removed
[lnd@NodeB mainnet]$ zcat lnd.log.22.gz | grep Removed
[lnd@NodeB mainnet]$ cat lnd.log | grep Removed
2024-01-24 02:46:35.127 [INF] HSWC: Removed 1 stale entries from network result store
2024-01-24 12:40:22.647 [INF] HSWC: Removed 1 stale entries from network result store

Thanks for taking the time to help. Appreciate it.

[bitromortac]

I have reproduced some of the issues and am working on a fix. I have a workaround, but this may be error prone for others, so I could give you advice on that on slack if you like.

[hazrulnizam]

Thank you for the follow up. Unfortunately, due to rushing, I couldn't wait and decided to spend a few thousand sats to close the only channel and proceeded to restore the node via aezeed seed to a clean PostgreSQL. So right now the node has a healthy graph after rebuilding from zero.

I do have a postgresql backup with the zombie edges, though, so I can test any workaround later after I'm done with the current project.

Therefore no need for slack assistance, I will wait for the workaround to be ready for public.