Strange statistics

[larsmans]

Reported by @dodijk: training on NL and inputting "UvA" produces

[{
        “target”:”Universiteit van Amsterdam”,
        "ngramcount”:22737,
        “linkcount":0,
        “commonness":1,
        “senseprob":0.0032546070281919337,
        “offset":0,
        “length":3
}]

Old semanticizer produced

{
        text: "UvA",
        links: [{
                linkOccCount: 79,
                text: "UvA",
                linkProbability: 0.2608695652173913,
                linkDocCount: 72,
                occCount: 375,
                id: "14815",
                senseProbability: 0.2608695652173913,
                senseOccCount: 79,
                title: "Universiteit van Amsterdam",
                url: "http://nl.wikipedia.org/wiki/Universiteit%20van%20Amsterdam",
                label: "UvA",
                senseDocCount: 72,
                priorProbability: 1,
                docCount: 276
        }],
        request_id: "9268bbee-e81e-463e-b155-3a8db256d171"
}

The ngramcount is probably too high, the sense probability too low.

[larsmans]

... and the link count should not be zero, or we shouldn't get that result at all.

Trivially fixed in dcf9186.

[larsmans]

Since the sense probability is derived from the ngramcount, the latter is probably off. Might be collisions in the count-min sketch.

[larsmans]

Branch fix-collision gives

[
  {
    "target": "Universiteit van Amsterdam",
    "ngramcount": 16244,
    "linkcount": 69,
    "commonness": 1,
    "senseprob": 0.004247722235902487,
    "offset": 0,
    "length": 3
  }
]

Still too high...

[larsmans]

15496 with a larger count-min sketch...