How to preprocess text for CTM?

[aneesha]

I read CTM uses both the preprocessed text for BOW and full text for BERT embedding. How can I create this as Dataset for the CTM model? Does saving an a OCTIS datasets automatically do this?

Many thanks

[silviatti]

Hi,
at the moment there is no way to do that in OCTIS. The modification would not require too much effort, but we need to think about the format of the saved file that represents the corpus (that is currently a .tsv file with some columns that may be optional). And so it would also require to re-preprocess the available datasets in this new format. Otherwise, this can be just an additional file (similar to the vocabulary file).

I am open to discuss this point.

Silvia

[lfmatosm]

Hi @silviatti. So, if I understand correctly, currently there's no way to load the unprocessed corpus documents on OCTIS' CTM while using its optimizer, in a manner similar to the one done on standalone CTM's README?

[silviatti]

I'm closing this issue because the discussion has moved here: #46