fit_report table: use tabulate and consistent value/uncertainty formatting?

[jagerber48]

Note that this discussion is related to the overall-width-formatting sciform issue but it is different. That issue is about if sciform should expose fixed width formatting functionality for users. This discussion is mainly about my recommendation for lmfit to use tabulate and standardized value/uncertainty formatting when generating fit reports.

Consider the example_fit_with_bounds.py example script. It outputs a fit report that looks like:

[[Fit Statistics]]
    # fitting method   = leastsq
    # function evals   = 79
    # data points      = 1500
    # variables        = 4
    chi-square         = 11301.3646
    reduced chi-square = 7.55438813
    Akaike info crit   = 3037.18756
    Bayesian info crit = 3058.44044
[[Variables]]
    amp:     13.8904759 +/- 0.24410753 (1.76%) (init = 13), model_value = 14
    period:  5.44026387 +/- 0.01416106 (0.26%) (init = 2), model_value = 5.4321
    shift:   0.12464389 +/- 0.02414210 (19.37%) (init = 0), model_value = 0.12345
    decay:   0.00996363 +/- 2.0275e-04 (2.03%) (init = 0.02), model_value = 0.01
[[Correlations]] 
  +----------+----------+----------+----------+----------+
  | Variable | amp      | period   | shift    | decay    |
  +----------+----------+----------+----------+----------+
  | amp      | +1.0000  | -0.0700  | -0.0870  | +0.5757  |
  | period   | -0.0700  | +1.0000  | +0.7999  | -0.0404  |
  | shift    | -0.0870  | +0.7999  | +1.0000  | -0.0502  |
  | decay    | +0.5757  | -0.0404  | -0.0502  | +1.0000  |
  +----------+----------+----------+----------+----------+

Here is the generating code. I see two issues with this table

1. The value/uncertainty formatting does not follow the BIPM best practices for presenting value/uncertainty pairs. In particular, it is typical to round the uncertainty and display only 1 or 2 significant figures. The value should be rounded so that the least significant digit displayed for the value matches up with the least significant digit displayed for the uncertainty. This means 13.8904759 +/- 0.24410753 should instead be displayed as 13.89 +/- 0.24. I argue the additional significant digits clutter the display and make it hard for the reader to quickly glean the important information from the table. For examples and some more info see BIPM guide to the expression of uncertainty in measurement section 7.2.2.
2. Looking at the source code in printfuncs.py, it is clear that lmfit is going through pains to get the whitespace and numeric widths right so that the resulting fit report table comes out aligned. However, even with those pains we see misalignment in the later columns of the variables table. The issues here are (a) the source code is complex because it needs to handle whitespace and (b) there are still remaining alignment issues.

The issues in point 2 can be easily solved by using the tabulate package which is designed to address exactly this issue. tabulate will handle all the white space and table construction (including row and column separators etc.) under the hood. The user only needs to provide an array of the input data to be displayed. I don't see a downside to adopting tabulate for lmfit reports.

The issues in point 1 can be addressed by using ufloat formatting from the uncertainties package (already a dependency of lmfit) or by using value/uncertainty formatting from the sciform package. I think either of these would be a net win for the lmfit fit report. However, there would be changes to how the numbers are displayed. First off, less significant figures would be shown for both the value and the uncertainty. Second, the widths of the numerical parts of the strings (that is excluding prepended/appended white space) would no longer exactly match. In my personal opinion these objections are not problematic, but I think this is a matter of personal taste so I'm open to other views.

Here is how the fit report would look using tabulate + uncertainties and still using gformat:

[[Fit Statistics]]
    ------------------  ----------
    # fitting method    leastsq
    # function evals    79
    # data points       1500
    # variables         4
    chi-square          11301.3646
    reduced chi-square  7.55438813
    Akaike info crit    3037.18756
    Bayesian info crit  3058.44044
    ------------------  ----------
[[Variables]]
     Name          Value          Percent Uncertainty    Constraint    Init Val    Model Val
    ------  -------------------  ---------------------  ------------  ----------  -----------
     amp    (1.389+/-0.024)e+01          1.76%              Vary          13          14
    period  (5.440+/-0.014)e+00          0.26%              Vary          2         5.4321
    shift    (1.25+/-0.24)e-01          19.37%              Vary          0         0.12345
    decay    (9.96+/-0.20)e-03           2.03%              Vary         0.02        0.01
[[Correlations]] 
    ┌────────┬─────────┬──────────┬─────────┬─────────┐
    │        │ amp     │ period   │ shift   │ decay   │
    ├────────┼─────────┼──────────┼─────────┼─────────┤
    │ amp    │ +1.0000 │ -0.0700  │ -0.0870 │ +0.5757 │
    ├────────┼─────────┼──────────┼─────────┼─────────┤
    │ period │ -0.0700 │ +1.0000  │ +0.7999 │ -0.0404 │
    ├────────┼─────────┼──────────┼─────────┼─────────┤
    │ shift  │ -0.0870 │ +0.7999  │ +1.0000 │ -0.0502 │
    ├────────┼─────────┼──────────┼─────────┼─────────┤
    │ decay  │ +0.5757 │ -0.0404  │ -0.0502 │ +1.0000 │
    └────────┴─────────┴──────────┴─────────┴─────────┘

The effect of gformat is to keep the final 4 floats in the Fit Statistics section to be the same character length.

Here is how the table might look using tabulate + sciform (and not using gformat):

[[Fit Statistics]]
    ------------------  -------
    # fitting method    leastsq
    # function evals    79
    # data points       1500
    # variables         4
    chi-square          11300
    reduced chi-square  7.554
    Akaike info crit    3037
    Bayesian info crit  3058
    ------------------  -------
[[Variables]]
     Name         Value         Percent Uncertainty    Constraint    Init Val    Model Val
    ------  -----------------  ---------------------  ------------  ----------  -----------
     amp      13.89 ± 0.24             1.76%              Vary          13          14
    period    5.440 ± 0.014            0.26%              Vary          2         5.4321
    shift     0.125 ± 0.024           19.37%              Vary          0         0.12345
    decay   0.00996 ± 0.00020          2.03%              Vary         0.02        0.01
[[Correlations]] 
    ┌────────┬─────────┬──────────┬─────────┬─────────┐
    │        │ amp     │ period   │ shift   │ decay   │
    ├────────┼─────────┼──────────┼─────────┼─────────┤
    │ amp    │ +1.0000 │ -0.0700  │ -0.0870 │ +0.5757 │
    ├────────┼─────────┼──────────┼─────────┼─────────┤
    │ period │ -0.0700 │ +1.0000  │ +0.7999 │ -0.0404 │
    ├────────┼─────────┼──────────┼─────────┼─────────┤
    │ shift  │ -0.0870 │ +0.7999  │ +1.0000 │ -0.0502 │
    ├────────┼─────────┼──────────┼─────────┼─────────┤
    │ decay  │ +0.5757 │ -0.0404  │ -0.0502 │ +1.0000 │
    └────────┴─────────┴──────────┴─────────┴─────────┘

Here sciform is used to present the last 4 floats in the Fit Statistics section to 4 significant figures. Note that each one has 4 significant figures but the overall character lengths of the numbers vary.

For both sciform and uncertainties, the value/uncertainty pair formatting could be adjusted using the various options available in the package. For example sciform can be used with paren_uncertainty=True and superscript=True to yield

[[Variables]]
     Name       Value       Percent Uncertainty    Constraint    Init Val    Model Val
    ------  -------------  ---------------------  ------------  ----------  -----------
     amp    1.389(24)×10¹          1.76%              Vary          13          14
    period  5.440(14)×10⁰          0.26%              Vary          2         5.4321
    shift   1.25(24)×10⁻¹         19.37%              Vary          0         0.12345
    decay   9.96(20)×10⁻³          2.03%              Vary         0.02        0.01

One could imagine the user passing in configuration options into the lmfit report_fit() function to control this display. e.g. for uncertainties the user could provide an uncertainties compatible format specifier and for sciform the user could provide a sciform Formatter (or sciform **kwargs that can be used to build a Formatter).

---

@newville or others: Are these two suggestions (consistent value/uncertainty formatting or tabulate adoption) of interest to lmfit? If so I would be happy to help make some of these changes. On my fork of lmfit I have a PR with a preliminary version of the tabulate + uncertainties + gformat version of the code: jagerber48#1. I also have a branch with the tabulate + sciform version.

I haven't yet checked if there are other places lmfit might benefit from tabulate.

[reneeotten]

I am not a fan of any of the suggested changes here. Admittedly I haven't spend the time to read the linked thread completely - too long 😉. Nothing shown here looks more readable to me than what we have now, but of course that's my personal opinion.

Sure, one should use a reasonable number of significant digits when reporting fitted parameters in any "official" report. However, an enduser should be able to so her/himself.

Also, if there is no need to depend in an external dependency as is the case here,we (or at least I) would rather not.

[jagerber48]

@reneeotten Thanks for your feedback. This is the type of feedback I was looking for.

• Don't worry about reading the other thread. The discussion here is self-contained.
• Thanks for sharing your opinion that the alternative options presented do not look more readable to you.
• I hear your point that for an "official" report the end user can/should do their own formatting to whatever standards they find suitable. But why not raise the standards of lmfit formatting so that it becomes more likely endusers could just use lmfit reports directly in "official" reports? It sounds like you are suggesting here that it is not important to you that lmfit conforms to official formatting standards such as the BIPM standards in the fit reports.
• Regarding the external dependency: Ignoring the sciform suggestion and considering the uncertainties + tabulate option, I guess you're talking about tabulate being a new dependency since uncertainties is already a dependency. Fair enough. Like I said in the original post, there are still some misalignments in the fit report. The trade off would be these misalignments could be eliminated for the end user and the printfuncs.py module would become less burdened by whitespace/tabulation management at the cost of including tabulate as an external dependency.

[newville]

@reneeotten I agree with everything you say.

I can appreciate wanting to follow the BIPM recommendations for how many significant digits to report for uncertainties. It is a reasonable option. We have no obligation to follow it or any other reporting guidelines in the standard reports we make. We are not preventing anyone from following those guidelines or making it hard for them to do so. Like, we have a standard-ish report at all. I am not sure about other packages, but as a prominent example, scipy.optimize does not. They make it hard for the user to know what the uncertainties are! So, we are not causing any problems, and we are not ignoring official recommendations -- that is utter nonsense and actually sort of off-putting.

But, I can see that helping users follow such guidelines could be helpful in some situations. Not at all urgent. Maybe an option, then again, maybe not.

While tabulate looks like an interesting package, I think we have a very limited number of tables we produce. I am not sure that I see the advantage of adding that dependency here.

[jagerber48]

@newville thanks again for this feedback. Yes, I agree lmfit doesn't present any roadblocks to standardized formatting. As you point out, it in fact makes it much easier to grab value/uncertainties from fits than scipy does. This is one of the main selling points for lmfit for me. The user can then do whatever they want with those numbers, including format them into a report however they like, using whatever tools they like.

But, I can see that helping users follow such guidelines could be helpful in some situations. Not at all urgent. Maybe an option, then again, maybe not.

Yes, I think helping users follow the guidelines would be a big selling point. I agree that it's not urgent, especially given, as you both point out, users can already easily take lmfit output and format their own tables. Just spelling out the idea here.

While tabulate looks like an interesting package, I think we have a very limited number of tables we produce. I am not sure that I see the advantage of adding that dependency here.

Fair enough. It's up to the maintainers whether the benefits from adding a new dependency offset the costs.

---

I'll leave it here for now unless other lmfit maintainers would like to weigh in, or if, in the future, minds are changed and it is decided that lmfit should support standardized value/uncertainty formatting or would benefit from using the tabulate package. In that event I would be happy to put together a PR.