Skip to content

Commit

Permalink
Updated benchmarks and README
Browse files Browse the repository at this point in the history
  • Loading branch information
@jamarino
jamarino committed Jul 11, 2023
1 parent a8f7aba commit 333a799
Show file tree
Hide file tree
Showing 3 changed files with 96 additions and 33 deletions.
94 changes: 74 additions & 20 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,58 @@ tree.Query(150); // result is {1, 2, 3}
// note that result order is not guaranteed
```

## Performance TLDR;

See performance section further down for more details.

### Query performance

```mermaid
gantt
title Query performance - Queries/second (higher is better)
dateFormat X
axisFormat %s
section Quick
11366 : 0, 11366
section Light
7025 : 0, 7025
section Reference
1603 : 0, 1603
```

### Initialization time

```mermaid
gantt
title Initialization time - miliseconds (lower is better)
dateFormat X
axisFormat %s
section Quick
39 : 0, 39
section Light
23 : 0, 23
section Reference
344 : 0, 344
```

### Initialization memory allocation

```mermaid
gantt
title Initialization memory allocation - megabytes (lower is better)
dateFormat X
axisFormat %s
section Quick
32 : 0, 32
section Light
16 : 0, 16
section Reference
342 : 0, 342
```

## Trees

This package currently offers two different interval tree implementations - `LightIntervalTree` and `QuickIntervalTree` - the former being the most memory-efficient and the latter using a bit more memory in exchange for some significant performance gains. Read on for more details and benchmarks.
Expand Down Expand Up @@ -77,33 +129,33 @@ The following table contains the change in memory usage measured between loading

It is clear that both `LightIntervalTree` and `QuickIntervalTree` offer better memory efficiency on average, compared to `RangeTree`. Additionally, memory growth is much more stable. Only a few objects are allocated per tree, and these are mostly long-lived and don't require (immediate) garbage collection. As a result, loading a tree does not cause a large spike in memory use and GC collections.

### Load 300.000 sparse intervals
### Load 250.000 sparse intervals

| Method | TreeType | Mean | Allocated |
|------- |---------- |----------:|----------:|
| Load | light | 30.27 ms | 32 MB |
| Load | quick | 61.70 ms | 53 MB |
| Load | reference | 472.00 ms | 623 MB |
| Load | light | 22.87 ms | 16 MB |
| Load | quick | 39.24 ms | 32 MB |
| Load | reference | 344.34 ms | 342 MB |

Loading data into `LightIntervalTree` and `QuickIntervalTree` is not only quicker, but also allocates a lot fewer objects / less memory in the process. This means less work for the GC and reduces potential spikes in memory usage.

> Note: "Allocated" memory is different from memory usage. It describes to total amount of memory written, not how much was ultimately kept.
### Query trees of 100.000 intervals
### Query trees of 250.000 intervals

| Method | TreeType | DataType | Mean | Allocated |
|--------|-----------|----------|-----------:|----------:|
| Query | light | dense | 1,350.2 ns | 1,197 B |
| Query | light | medium | 273.4 ns | 197 B |
| Query | light | sparse | 158.4 ns | 59 B |
| Query | quick | dense | 435.7 ns | 1,197 B |
| Query | quick | medium | 152.3 ns | 197 B |
| Query | quick | sparse | 110.1 ns | 59 B |
| Query | reference | dense | 4,007.2 ns | 5,556 B |
| Query | reference | medium | 876.1 ns | 1,436 B |
| Query | reference | sparse | 428.1 ns | 908 B |

`LightIntervalTree` is about 3-4 times quicker to query. `QuickIntervalTree` manages 4-9 times faster queries, and pulls ahead in dense datasets.
| Query | light | dense | 142.35 ns | 107 B |
| Query | light | medium | 98.11 ns | 60 B |
| Query | light | sparse | 82.78 ns | 40 B |
| Query | quick | dense | 87.98 ns | 107 B |
| Query | quick | medium | 79.01 ns | 60 B |
| Query | quick | sparse | 72.18 ns | 40 B |
| Query | reference | dense | 623.72 ns | 1,256 B |
| Query | reference | medium | 458.71 ns | 996 B |
| Query | reference | sparse | 317.60 ns | 704 B |

`LightIntervalTree` is about 3-4 times quicker to query. `QuickIntervalTree` manages 4-7 times faster queries, and pulls ahead in dense datasets.

## Thread Safety

Expand All @@ -114,11 +166,13 @@ When using trees in a concurrent environment, please be sure to initialise the t

## TODO list

* Add method for querying a range
* Add remove methods
* Implement method for querying a range
* Implement remove methods
* Consider adding a new auto-balancing tree
* Consider adding constructors that take custom `Comparer`s
* Consider adding constructors that take a `capacity` hint
* Add constructors that take a `capacity` hint
* Add dotnet7 INumber<T> TKey constraint for improved performance (approx 2x query performance)
* Replace recursive methods with iterative ones where possible
* Experiment with interal data arrangement to improve indexing performance and data linearity

## Optimizations over RangeTree

Expand Down
16 changes: 10 additions & 6 deletions Tools/Benchmark/LoadBenchmarks.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,16 +14,20 @@ public class LoadBenchmarks
[ParamsSource(nameof(TreeTypes))]
public string TreeType { get; set; } = string.Empty;

[Params(250_000)]
public int IntervalCount = 1;

[GlobalSetup]
public void GlobalSetup()
{
const long RangeMin = 300_000_000_000;
const long RangeMax = 700_000_000_000;
var random = new Random(123123);
var random = new Random(123);

var max = 10 * IntervalCount;
var maxIntervalSize = 2 * max / IntervalCount;

_ranges = Enumerable.Range(0, 300_000)
.Select(i => random.NextInt64(RangeMin / 1000, RangeMax / 1000))
.Select(i => (i * 1000, (i * 1000) + (random.Next(1, 3) * 1000) - 1, random.Next(10_000)))
_ranges = Enumerable.Range(0, IntervalCount)
.Select(i => random.NextInt64(0, max))
.Select(i => (i, i + random.Next(1, maxIntervalSize), random.Next(10_000)))
.ToList();
}

Expand Down
19 changes: 12 additions & 7 deletions Tools/Benchmark/QueryBenchmarks.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ namespace Benchmark;
[MemoryDiagnoser]
public class QueryBenchmarks
{
const int IntervalCount = 100_000;
private Dictionary<string, IntervalTree.IIntervalTree<long, int>> _treeCache = new();
private Dictionary<string, IEnumerable<Interval<long, int>>> _dataCache = new();

Expand All @@ -19,6 +18,9 @@ public class QueryBenchmarks
[Params("sparse", "medium", "dense")]
public string DataType { get; set; } = string.Empty;

[Params(250_000)]
public int IntervalCount = 1;

public IntervalTree.IIntervalTree<long, int> GetLoadedTree(string treeType, string dataType)
{
var key = $"{treeType}:{dataType}";
Expand Down Expand Up @@ -46,48 +48,51 @@ public void GlobalSetup()
{
var random = new Random(123);

// approx 20% coverage
var sparse = Enumerable.Range(0, IntervalCount)
.Select(_ =>
{
var start = random.Next(10*IntervalCount);
var start = random.Next(25*IntervalCount);
return new Interval<long, int>(
start,
start + random.Next(1, 20),
start + random.Next(1, 10),
1);
})
.ToList();
_dataCache["sparse"] = sparse;

// approx 100% coverage
var medium = Enumerable.Range(0, IntervalCount)
.Select(_ =>
{
var start = random.Next(10*IntervalCount);
return new Interval<long, int>(
start,
start + random.Next(10, 200),
start + random.Next(1, 20),
1);
})
.ToList();
_dataCache["medium"] = medium;

// approx 500% coverage
var dense = Enumerable.Range(0, IntervalCount)
.Select(_ =>
{
var start = random.Next(10 * IntervalCount);
return new Interval<long, int>(
start,
start + random.Next(100, 2000),
start + random.Next(1, 100),
1);
})
.ToList();
_dataCache["dense"] = dense;
}

[Benchmark(OperationsPerInvoke = 10*IntervalCount)]
[Benchmark(OperationsPerInvoke = 10_000)]
public void Query()
{
var tree = GetLoadedTree(TreeType, DataType);
for (var i = 0; i < IntervalCount * 10; i++)
for (var i = 0; i < 10_000; i++)
{
tree.Query(i);
}
Expand Down

0 comments on commit 333a799

Please sign in to comment.