v1.17.6

v1.17.6

With this release, many hardening issues were tackled. The team's focus has been mostly on improving connectivity of nodes across the network and bringing performance improvements to chunk caching operations.

Also featured is a new DB command that will perform a chunk validation of the chunkstore, similar to the optional step in the compaction command.

The retrieval protocol now has a similar multiplexing capability like pushsync, where multiple, in parallel, requests are fired from a forwarder peer that can directly access the neighborhood of a chunk.

For questions, comments, and feedback, reach out on Discord.

Bug fixes

• Fixed a bug where parallel uploads can cause a race condition in the uploadstore. ( #4434 )

Features

• Added a new DB cmd that performs a validation check of the chunks in the chunkstore. ( #4435 )
• Added multiplexing to the retrieval protocol where a forwarding peer that can reach in to the neighborhood of the chunk fires multiple attempts to different peers. ( #4405 )

Hardening

• Added extra documentation about the logger API in the CODING.md. ( #4406 )
• Fixed logs containing wrong token name. ( #4408 )
• Added metrics for zero addressed chunks received by the pullsync protocol. ( #4407 )
• Kademlia depth value is overwritten by the storage radius for full nodes. ( #4410 )
• Salud response duration check is now more stricter. ( #4417 #4426 )
• Upgraded libp2p to the latest version v0.30.0. ( #3927 )
• When batches expire and are removed the batchstore, the stamp issuer data is also removed ( #4416 #4431 #4439 )
• Add a new log to display the amount of time the postage listener will sleep for until the next blockchain sync event ( #4444 #4426 )
• API now returns 404 instead of 500 when no peer can be found for a chunk retrieval attempt. ( #4436 )
• Upgraded crypto related packages. ( #4425 )
• Added various connectivity related improvements: ( #4412 )
  • The reachability of connected peers is tested periodically instead of once at the time of the initial connection.
  • All and not a small subset of neighbor peers are broadcast to a newly connected neighbor.
  • Neighbor peers are periodically broadcast to other neighbors.
  • A peer will be re-added to the addressbook if hive detects an underlay change.

Performance

• Added a cache eviction worker so cached chunks do not need to removed immediately when adding new entries to an over-capacity cache. ( #4423 #4433 )
• The POST /pins/{ref} API endpoint now stores chunks in parallel. ( #4427 )

For a full PR rundown please consult the v1.17.6 diff.

v1.17.5

v1.17.5

In this small but important release, the Bee team introduces a new db compaction command to recover disk space. To prevent any data loss, operators should run the compaction on a copy of the localstore directory and, if successful, replace the original localstore with the compacted copy. The command is available as a sub-command under db as such:

bee db compact --data-dir=

The pushsync and retrieval protocols now feature a fallback mechanism of trying un-reachable and un-healthy peers in the case that no reachable or healthy peers are left.

We've also added new logging guidelines for contributors in the readme.

For questions, comments, and feedback, reach out on Discord.

Bug fixes

• Fixed a bug where a node can get stuck syncing the same interval if the upstream peer is unable to send the chunk data. ( #4339 )

Features

• Added a new localstore compaction command that resizes sharky to the smallest size possible. ( #4329 )

Hardening

• Added a new logging guideline for contributors ( #4352)
• Improved logging of the retrieval pkg and increased the min healthy peers per bin in the salud service.
• Varying levels of peer filtering for the pushsync and retrieval protocols ( #4388 )

For a full PR rundown please consult the v1.17.5 diff.

v1.17.4

For the past few weeks, the Bee team's focus has been on improving network health, observability, and user experience.

Node operators, can now mine an overlay address for specific neighborhoods for fresh nodes by using the new --target-neighborhood option. The new Swarm Scanner neighborhoods page displays neighborhood sizes and is a great tool to be used in tandem with this new feature.

Uploads are now by default deferred, as they were before the v1.17.0 release.

Additionally, the default postage stamp batch type is now immutable.

Another behavioral change is that swap-enable is now by default false and the bee start command without additional options starts the node in ultra-light mode. Full node operators must enable the option with swap-enable: true if not already enabled for their nodes to continue to operate as normal.

We have also improved logging across many different services and protocols.
Pushsync and retrieval protocols now report error messages back to the origin node instead of the generic "stream reset" errors. As a result, the protocol version has been bumped, making this a breaking change. It is imperative that operators update their nodes asap.

Previously, nuking a node could cause syncing problems due to the fact that
syncing intervals were never reset. This issue has now been tackled by having nodes detect that a peer's localstore has been nuked. They are able to do this by comparing the peer's localstore epoch time across connections.

For questions, comments, and feedback, reach out on Discord.

Bug fixes

• Pullsync intervals are now reset when the peer's localstore epoch timestamp changes. ( #4290)
• Fixed the issue of not being able to pass a "0" parameter in the API. ( #4301)
• Fixed a bug where the context used for cachestore operations is canceled before a chunk can be cached. ( #4307 )

Features

• Mining an overlay address for specific neighborhoods. ( #4293 )
• Pushsync and retrieval protocol propagate the message of the error that terminates the request back to the origin peer. ( #4321 )
• The naked bee start cmd starts the node in ultra-light mode. ( #4326)

Hardening

• Logging improvements. ( #4295 #4296 #4297 #4302)
• The type of refCnt field of the chunkstore changed from uint8 to uint32, increasing its capacity. ( #4299 #4309 )
• Default batch type is now immutable. ( #4304 )
• The testnet bootnode address is now supplied by default in the configuration. ( #4317 )
• Uploads are now deferred by default. ( #4318 )

For a full PR rundown please consult the v1.17.4 diff.

v1.17.3

v1.17.3

The Bee team is excited to announce v1.17.3! 🎊

With this release, we introduce healthy peer filtering for retrieval requests to improve data download reliability and speed.

The behavior of the upload APIs are changed back to pre-1.17.0 so parallel uploads with the same batch are now allowed.

The deprecated bee-clef is now removed from all bee packaging and docker images so operators should update their setups accordingly.

The node during the initial bootup will go through a localstore recovery process that should take a few minutes to complete. A localstore leak was detected in the v1.17.0 release, so with this recovery process and the fix included in this release, disk space consumption should become stable.

Pullsync chunk intervals are reset to resync chunk data across all neighborhoods as the final step to bring the localstore release to a completely stable state.

The feeds API now features a start index hint to optimize the retrieval loop of the latest feed.

For questions, comments, and feedback, reach out on Discord.

Bug fixes

• Fixed a mutex bug in the localstore migration step. ( #4267)
• Fixed a light node bug where a panic is avoided when querying ReserveLastBinIDs on ultra-light node with no reserve. ( #4271)
• Fixed a localstore leak where sharky slots of evicted chunks were not marked as free for future chunks to be stored ( #4276 )

Features

• Added a new Feed API parameter to speed up feed lookups. ( #4244 )

Performance

• Parallel uploads with the same batch are supported again. ( #4263 )
• Retrievals only utilize healthy and reachable peers. ( #4164 )

Hardening

• Sync intervals are reset to resync reserves. ( #4257 )

For a full PR rundown please consult the v1.17.3 diff.

v1.17.2

With the last release we upgraded the mainnet to use the new storage layer. However, seeing the results from the redistribution game, we found some issues in it that were previously missed. The main problem we saw was that some of the pullsync indexes were being overwritten and hence, although the nodes have the chunks, they were not able to sync them with the network. This is now fixed, but we will need to do a migration to reconstruct the pullsync indexes on the nodes.

We also found that the cache implementation was over-complicated, which resulted in some bugs, so this has been simplified as well. There will be another smaller migration to update the cache indexes. Node operators are requested to update their setups as soon as possible. The migrations can take up to 20-30 mins depending on the setup, so please be prepared for this as well. We also strongly advise against concurrently upgrading nodes which are running on the same physical drive since it could drasticly increase migration time.

There is another simplification on the reserve eviction and postage batch expiration interactions. In the previous design, the salud health service would check whether reserve sizes (defined as the number of chunks in reserve at storage depth) were similar within some range of tolerance, and mark the node as unhealthy if the difference in reserve size surpassed that tolerance. In the current design, we don't expect the reserve sizes to match on neighbours as this size is now defined as the total number of reserve chunks on the node, regardless of depth. This can include chunks from lower storage depths which are not yet evicted. However, this should not stop the nodes from producing matching samples for the redistribution as we only consider the storage-depth and above chunks for the sample. To address this issue we have removed the reserve-size related checks in the salud service. Nodes will no longer mark themselves as unhealthy if their reserve sizes are off, and each node can now unreserve at its own speed.

New features

• Added isHealthy to redistributionstate (#4243).

Hardening

• Simplify batch expiration and unreserve interaction. (#4245)
• Simplify cache handling. (#4242)
• Remove reserve size self-health check. (#4255)

Bugfixes

• BinID handling fixed. (#4248)
• Synchronize chunkstore invasive operations. (#4253)

For a full PR rundown please consult the v1.17.2 diff.

v1.17.1

This is a patch release on top of the 1.17.0 release.

The main fix in this patch is the storage radius is bootstrapped manually using the current value from the mainnet, and not through any other mechanism.

Refer to the main release notes here.

v1.17.0

Bee v1.17.0 is finally here. This release will be the most significant update to the bee client in a long time and will be a breaking release due to protocol and API changes. The Bee team has been working on the new storage layer for quite some time now. The new storage layer is the repayment of a massive technical debt for the team as the old local storage component was not in a good state and was no longer maintainable. Follow all the communication on our discord to know all the details!

The goal of this upgrade is not for any performance benefits at the moment, but to ensure all existing functionality is retained and the storage component is simplified and hence improving maintainability.

The API changes are mostly consolidations and minor changes to the response types. Users are advised to go through the API docs and update their libraries/tooling accordingly.

As mentioned earlier, this will be a breaking release on the protocols-side as well. Pullsync, hive, retrieval, and handshake versions will be bumped. As a result, all node operators are requested to update their nodes to the latest release as soon as possible. The nodes will also do a full resync of the postage stamp contract events to properly flush out all expired content from their reserves once they upgrade to the new storage layer.

The upgrade, when first deployed, will begin to migrate data from the old localstore to the new and in the best case should finish in a few minutes but may take longer on slower devices. Because data migration is involved, it is higly advised to back up everything before the upgrade.

Warning for multiple node operators: the upgrade should be done by one node at a time for nodes that share the same physical disk as the initial migration from the old localstore to the new may take up to 30 minutes on older/slower hardware.

New features

• Support multiple stamps on chunks. Chunks uploaded with different stamps will be tracked separately in the network. This means the chunk will only be removed from the network once all the stamps associated with it are gone.
• Stewardship PUT endpoint now allows users to configure a stamp to use for re-upload. This would allow users to re-upload their pinned content with new stamps if the old one was expired.

Breaking changes

• /tags endpoint is now consolidated on the main API. The debug API tags endpoint is deprecated. Also, the tag response fields have changed.
• DELETE /chunk endpoint is also deprecated. This endpoint was mainly present for testing and will no longer be supported.

Hardening

• Pinning data is now managed separately. As a result the pinned and reserve content will no longer affect each other.
• Self-health check based on reserve size of neighborhood. (#4110)
• Synchronize interval state storing. (#4115)
• Prune out-of-depth oversaturated bins based on peer health. (#4092)
• Cleanup mutex handling in batchstore and reserve. (#4159)

Bugfixes

• Unreserve batch incorrectly unpins chunks. Also other issues related to pinning. (#3037, #3224, #3302, #3514)
• Reduce light-node disk usage. (#2990)
• Reuploading content without increasing bucket collision counters. (#2851)
• Upload related memory leak. (#4009)

For a full PR rundown please consult the v1.17.0 diff.

v1.16.1

This one is a patch release to address a bug found in the new salud service.

Bug fixes

• Fixed a panic in the salud service when the response to the status request is terminated early by the peer. ( #4077)

v1.16.0

v1.16.0

The bee team is excited to announce v1.16.0! 🎉

The team has been busy researching and testing ways to help swarm remain a reliable and healthy network, and to that end, we are happy to announce a brand new health service, salud.

With salud, nodes will periodically perform certain health checks on it's connected peers with data acquried from the status protocol. The checks as of this release are based on the duration of response to the status protocol msg, number of connected peers, the storage radius, and total batch commitment as computed by each peer.

For duration and number of connected peers, each peer must be within the 80th percentile to be deemed as healthy. Radius and batch commitment are measured in terms of the most common values as reported by each connected peer. Measurements are created anew for each peridoic health check.

A self check is also in place where if the node's own storage radius does not match with the rest of the networks', the node won't participate in the schelling game.

With this release, only the pushsync protocol utilizes the filtering of peers for requests based on the status of health.

For questions, comments, and feedback, reach out on discord.

Breaking changes

• The status protocol now includes the reachability of a peer and the total batch commitment of the network reported by each peer (#4054, #4042)

Bug fixes

• Fixed the last played round field in the redistribution state endpoint to report the corrent round instead of the previous round. ( #4062)

Hardening

• Loosed the requirements of the pushsync protocol so that in the case a neighborhood is down or does not exist, the chunk is pushed as close as possible to it's neighborhood. (#4064)
• Increased the wake up interval for the depthmonitor from 5 to 15 minutes. (#4068)

For a full PR rundown please consult the v1.16.0 diff.

v1.15.0

v1.15.0

The bee team is excited to announce v1.15.0!

With this release, we introduce a new pushsync feature to improve chunk syncing and replication in the network during uploading. Peers that forward chunk into the neighborhood will fire multiple requests to target multiple storer nodes. Forwarding will also terminate at the first peer within the neighborhood and will no longer be forwarded to the most closest peer within the network.

As a result of the change in protocol logic, the protocol version has been bumped so it's important that you upgrage your nodes to the latest version.

We've added two new fields to the status protocol response: total amount from the chainstate and mode of operation of the peer (light or full mode).

For questions, comments, and feedback, reach out on discord.

Breaking changes

• Pushync forwarding nodes one hop from the neighborhood multiplexes the chunk to multiple storer nodes in the neighborhood. Forwarding terminates at the first neighbor. (#4008)

Hardening

• Blocklist endpoint now correctly display the mode and reason of the blocklisting of the peer. (#4015)
• Storage radius decreasing based on reserve size increased back to 50% threshold. (#4013)
• Chainstate total amount and peer mode added to the status protocol. (#4002 #4010)
• Connecting light nodes are not rejected based on bin size. (#4018)

Perfomance Improvements

• Initial historical syncing is limited to reduce workload. (#4000)
• Peer skiplist mechnism in retrieval and pushsync protocols now prune peers in a much more effiecient way. (#4001)

For a full PR rundown please consult the v1.15.0 diff.