whisper grammar: experimental implementation with boost::spirit

[fraxy-v]

An explorative re-implementation of @ejones's parser which IMHO captures the advantages and disadvantages of boost::spirit very well.

[ulatekh]

I'm interested in alternate grammar implementations, since the existing one has odd accuracy issues, especially when it comes to matching whole words from the grammar. But what does this branch do? It doesn't seem to substitute the boost::spirit implementation in place of the existing grammar implementation.

[fraxy-v]

True. It doesn't replace the original, but adds a test_parse method and its implementation in test-grammar-parser.cpp, and a bit of a hand-wavy test test-grammar.cpp which compares the new boost::spirit implementation against the old one. That's the raw stage that I left it in, since I just wanted to see how it would work with boost::spirit.

I guess you can just plug an example input exhibiting these issues into the test code and see if boost::spirit behaves differently and, if it doesn't, if it can be easily fixed. Let me know if you need help with that.

[fraxy-v]

Having said that, I would also say it may be better to fix the problem in the original implementation. I am fairly familiar with the grammar parser, so If you give me an example grammar input demonstrating the oddities, I may be able to suggest a fix.

[ulatekh]

The only grammar I have is a very large one. If I can figure out how to cut it down to where it reproduces the problem, I will. I imagine the problem is that whisper.cpp's grammar is character-based, and if it finds multiple possibilities that begin with the same prefix, but can't decide between them, it terminates recognition at the common prefix, instead of realizing that an incomplete word is not a match.

[fraxy-v]

Okay, but that doesn't sound like a parser bug, unless you have some modification in mind.

[ulatekh]

It's a bug in how whisper.cpp applies grammars, though. It does me no good to get a response back from Whisper that doesn't match the grammar I gave it.

[fraxy-v]

Fair enough, but the application of the grammar is terra incognita for me. Want to help, but I don't know if I could. @ggerganov what do you think?

[ggerganov]

A repro would be needed - without a specific example of the problem, I don't think we can do anything