Implement Chi-Squared Testing, Normalized Estimation Error Squared (NEES) and Normalized Innovation Squared (NIS) for tuning the Kalman filter

[ChristopherRabotin]

High level description

This feature adds functionality to perform Chi-Squared tests, NIS and NEES analysis on Kalman filter estimators.

The Chi-Squared test is a statistical hypothesis test that is used to compare the observed and expected values of a variable. The NEES is a measure of the accuracy of a Kalman filter estimate. This feature allows users to compare the NEES of two different Kalman filter estimators using the Chi-Squared test. Additionally, the NEES can be used to diagnose problems with a Kalman filter estimate. This feature provides a useful tool for evaluating the performance of Kalman filter estimators.

The main difference between NIS and NEES is that NIS is a measure of the accuracy of the innovation signal, while NEES is a measure of the accuracy of the state estimate.

The innovation signal is the difference between the measurement and the predicted measurement. The state estimate is the Kalman filter's best guess at the state of the system.

A low NIS indicates that the Kalman filter is able to accurately predict the measurement. A low NEES indicates that the Kalman filter is able to accurately estimate the state of the system.

Both NIS and NEES are useful tools for evaluating the performance of a Kalman filter. However, NIS is more sensitive to the accuracy of the innovation signal, while NEES is more sensitive to the accuracy of the state estimate.

Some implementations of a chi squared test:

• https://github.com/ChristopherRabotin/gokalman/blob/master/chisquare.go
• https://github.com/ChristopherRabotin/gokalman/blob/master/montecarlo.go
• https://github.com/ChristopherRabotin/gokalman/blob/master/montecarlo_test.go (includes the Chi Square tests)

Original issues:

• https://gitlab.com/nyx-space/nyx/-/issues/195
• https://gitlab.com/nyx-space/nyx/-/issues/193

Requirements

1. Calculate NIS and NEES for each measurement during the Kalman filter estimation process.
2. Compare NIS and NEES against some user-defined thresholds to detect if the Kalman filter is under-performing.
3. If the NIS or NEES exceeds the user-defined thresholds, log a warning message to the user.
4. Provide an option for the user to specify the thresholds for NIS and NEES.
5. Allow the user to customize which specific measurements should be evaluated for NIS and NEES (e.g., range, Doppler, etc.).
6. Provide documentation on how to interpret the results of the NIS and NEES evaluation.

Test plans

Validation of NIS and NEES values

• Done

Input: A set of measurements and corresponding state estimates
Action: Calculate the NIS and NEES values for each measurement and state estimate pair
Expected Output: NIS and NEES values that are within acceptable ranges for accurate Kalman filtering

Comparison of NIS and NEES values for different Kalman filter configurations

• Done

Input: A set of measurements and corresponding state estimates processed by different Kalman filter configurations
Action: Calculate the NIS and NEES values for each measurement and state estimate pair for each configuration
Expected Output: Comparison of NIS and NEES values to determine the most accurate Kalman filter configuration

Diagnosis of Kalman filter performance using NIS and NEES

• Done

Input: A set of measurements and corresponding state estimates processed by a Kalman filter
Action: Calculate the NIS and NEES values for each measurement and state estimate pair, and analyze the results to diagnose any performance issues
Expected Output: Identification of any issues with Kalman filter performance and recommendations for adjusting the filter parameters to improve accuracy

Sensitivity analysis of NIS and NEES to process and measurement noise covariance

• Done

Input: A set of measurements and corresponding state estimates processed by a Kalman filter with varying process and measurement noise covariance values
Action: Calculate the NIS and NEES values for each measurement and state estimate pair for each covariance value, and analyze the results to determine the sensitivity of NIS and NEES to these parameters
Expected Output: Identification of the optimal process and measurement noise covariance values for accurate Kalman filtering, based on the sensitivity analysis results.

Test for significant difference between two NEES values:

• Done

1. Set up two Kalman filter estimators with different process and measurement noise assumptions, but using the same input data.
2. Compute the NEES values for both estimators.
3. Calculate the chi-squared value for the two NEES values using a chi-squared test.
4. If the chi-squared value is significant (i.e. the p-value is below a certain threshold), then the two NEES values are significantly different, indicating that the two estimators have different accuracy.
5. Otherwise, the two NEES values are not significantly different.

Test for high NEES indicating problem with Kalman filter:

1. Run a Kalman filter on simulated data, with known process and measurement noise parameters.
2. Inject some noise into the measurement data to create a "bad" measurement.
3. Compute the NEES value for the "bad" measurement.
4. Calculate the chi-squared value for the NEES value using a chi-squared test, comparing it to a chi-squared distribution with appropriate degrees of freedom.
5. If the chi-squared value is significant, then it indicates that the NEES value is significantly higher than expected, which may indicate a problem with the Kalman filter parameters or assumptions.
6. Otherwise, the NEES value is not significantly different than expected, indicating that the Kalman filter is performing adequately.

Design

Document, discuss, and optionally upload design diagram into this section.

Algorithm demonstration

Dr. Ahmed has an arXiv paper on NEES: https://arxiv.org/pdf/1807.08855.pdf .

API definition

Define how the Nyx APIs will be affect by this: what are new functions available, do any previous function change their definition, why call these functions by that name, etc.

Try to add an ASCII diagram of how this should work.

Detailed design

The detailed design will be used in the documentation of how Nyx works.