Allow integration of custom search terms

[tim-janik]

The Anklang documentation is generated via Poxy + m.css, it's internal APIs are implemented in C++ and JS. Since Doxygen has no JS support, the JS docs are generatred via a jsdoc2md.js script + pandoc. The HTML output is included into the Doxygen docs with @file f.js \n @htmlinclude[block] f.js.html. This allows browsing of the JS docs within the generated documentation, but no searches.

Ideally, I would be able to supply a file with a list of additional search tokens, that is supported for search in addition to the C++ search terms. A corresponding feature request is added to Poxy: Poxy #23

For the time being, I have added a list of extra searchable tokens to searchdata-v2.js and extended the search-v2.js:searchAndRender function to also search the extra tokens. It's a hack, that ideally could be replaced by a more idiomatic solution in the future. The current approach e.g. doesn't support auto completion for JS token. It can be tested out here:
https://tim-janik.github.io/docs/anklang/util_8js.html#search

[mosra]

The most ideal way would of course be to support JS directly, and without Doxygen in the loop :) So, for example, using JSDoc to parse the JS code + comments and give the parsed output back in some JSON, which would then be processed to a template that actually matches JS features (instead of trying to fit it into the C++-oriented template). Given this wouldn't require undoing ugly Doxygen bugs, and assuming JSDoc gives out something already usable, I think it could be quite manageable. The Python doc generator is made from scratch as well, and the main complexity there was mostly about type annotations and lacking introspection/reflection capabilites, something I think you wouldn't run into with JSDoc.

What's already builtin is a feature allowing you to specify additional keywords with @m_keyword{} and @m_keywords{}, Poxy should support those commands directly without having to edit any Doxyfile. So, at the very least, you could add the JS symbols as keywords to the page, so searching for them would lead you to the page/file itself. But just the page/file, not an anchor, and there wouldn't be a way to distinguish functions from objects etc.

/**
@file util.js

@m_keywords{vue_mixins}

...
*/

Another option, and quite easy implementation-wise, could be to expose a config option with a list of tokens that get fed directly into the search data in state.search and processed in build_search_data(). The configuration in m.css can be a Python script so it'd be for example

SEARCH_EXTRA = [
   # name, url, type
   ('vue_mixins', 'util_8js.html#vue_mixins', 'CLASS'),
   ('vue_export_from_class', 'envue_8js.html#vue_export_from_class', 'FUNC'),
   ...
]

Given the config is scriptable, it shouldn't be too laborous, however it's still pretty much tied to the set of type IDs used for the Doxygen symbols (so no "module", for example), to various C++-isms, and is quite low-level compared to what a builtin JS generator could do. A minimal (untested) patch to allow this would be something like this:

diff --git a/documentation/doxygen.py b/documentation/doxygen.py
index 2d5a6f9a..ecc506fb 100755
--- a/documentation/doxygen.py
+++ b/documentation/doxygen.py
@@ -145,6 +147,7 @@ copy a link to the result using <span class="m-label m-dim">⌘</span>
 """,
     'SEARCH_BASE_URL': None,
     'SEARCH_EXTERNAL_URL': None,
+    'SEARCH_EXTRA': [],
 
     'SHOW_UNDOCUMENTED': False,
     'VERSION_LABELS': False
@@ -2501,8 +2504,18 @@ def build_search_data(state: State, merge_subtrees=True, add_lookahead_barriers=
     def strip_tags(text):
         return strip_tags_re.sub('', text)
 
+    search = state.search.copy()
+    for name, url, type in state.config['SEARCH_EXTRA']:
+        result = Empty()
+        result.flags = ResultFlag.from_type(ResultFlag(0), EntryType[type])
+        result.url = url
+        result.prefix = []
+        result.name = name
+        result.keywords = []
+        search += [result]
+
     symbol_count = 0
-    for result in state.search:
+    for result in search:
         # Decide on prefix joiner. Defines are among the :: ones as well,
         # because we need to add the function macros twice -- but they have no
         # prefix, so it's okay.

Would it make sense to be done this way? For Poxy AFAIK there's a TOML configuration file so this would have to be propagated through from there somehow, which might hinder the scriptability aspect. Or @marzer do you have a way to "pass through" a python script directly to a m.css conf.py?

[marzer]

Or @marzer do you have a way to "pass through" a python script directly to a m.css conf.py?

Not currently, but would be pretty easy to expose. There's been a few debugging situations where it would have been useful for me, too 😅