Enable SQL engine without flex and Bison installed

[evanmiller]

Flex and Bison are only really needed by package maintainers to generate new C files. So check in the generated files, enable SQL by default, and allow disabling with --disable-sql.

Fixes #273. Supersedes #276.

[rhurlin]

Hi Evan,
Thanks for these changes!

Do we even need dependencies like lexer, yacc and bison anymore, are they still useful?

I am asking as the maintainer of the FreeBSD port of MDB Tools ;)

[evanmiller]

@rhurlin Those dependencies were only ever needed at compile-time. With these changes, they'll only be needed when a developer is modifying the lexer or parser.

[rhurlin]

@evanmiller OK, so it is enough to have them installed on the developer box (is mostly the case there), the end user will do without.

Thanks for the explanation :)

[nirgal]

I'm not sure this is a good idea.
We could also commit .exe ... but the sources are much better.
@rhurlin I'm not sure about BSD, but Debian forbid the use of generated files. I would definitely discard these files if I were you, and generate them from source.

[rhurlin]

@rhurlin I'm not sure about BSD, but Debian forbid the use of generated files. I would definitely discard these files if I were you, and generate them from source.

@nirgal I'm not sure if my skills are sufficient to actually understand the changes.

As far as I can see, lexer and parser are still used under src/sql/ even though src/sql/Makefile.am now only contains AM_YFLAGS -d (without -o parser.c).

I must confess, I don't even have knowledge about what lexer.l and parser.y do here when building mdb-sql and libmdbsql.so :(

[evanmiller]

This page might help clarify the new behavior:

https://www.gnu.org/software/automake/manual/html_node/Yacc-and-Lex.html

I can remove the generated files from the PR's source tree; but according to the above, generated files will still be included in distribution tar balls.

[rhurlin]

@evanmiller @nirgal I am beginning to understand how the "mechanics" of lex and yacc work via automake. Thanks for the explanations and critical advice.

On FreeBSD, the port so far is such that flex and bison are available and the C files are compiled.

If I compare the files lexer.c, parser.h and parser.c built under FreeBSD with the ones provided according to this Issue #296, there are numerous differences, especially in lexer.c. Among them are also different data types, indexing and casts.

You will find diff's of all three files attached:
yacc_lex_FreeBSD.tar.gz

I cannot judge to what extent these differences are important for the respective platform (here FreeBSD).

[evanmiller]

@rhurlin Likely the differences are just due to different versions of flex generating the files.

[rhurlin]

@evanmiller I already hoped so :)

If this is true, there should be no difference in the interpretation of SQL code between MDB Tools with and without C files included. The future will show that ...

[evanmiller]

One difficulty of excluding the files from the source tree is that configure will need to check for bison and flex only if the generated files are not present (e.g. after cloning the repository). I'm not sure how to thread the needle to check for the right tool versions on fresh checkouts but skip the tool checks on the distributed tar balls.

[rhurlin]

By "distributed tar balls" do you mean the four files listed under Assets on Github? I have no experience with them, as FreeBSD basically downloads the sources and compiles with them.

Or does this mean ready-made packages that are offered on distros like Windows, Linux, macOS, etc.?

Sorry for my little knowledge about such workflows :(

[evanmiller]

@rhurlin "Distributed tar balls" refers to the Asset files mdbtools-0.9.x.tar.gz and mdbtools-0.9.x.zip. These are created by myself for each release using make dist, and are manually uploaded to GitHub. The purpose of this PR is to remove the flex and bison dependencies for users who obtain the code in this way, since these tools are not actually required at run-time.

[rhurlin]

@evanmiller Ah! Now I think I understand the meaning of the assets (tar balls) better. Thanks for the explanation.

Unfortunately I have little experience with automake and writing configure scripts, so can't help with threading the needle, sorry.