Search Engine - Rebuild Index does not work

[skyreacher]

Hello,

I've got a couple of Wiki.js instances running in a Kubernetes cluster. Wiki.js instances are started with HELM and PostgreSQL is used as database.
So far everything works fine. I've been using search engine "Database - PostgreSQL" which works fine in most cases.

But I have one instance of Wiki.js, which doesn't succeed to build an index. This results in searches non returning any hits.

In the pod's logs I see this:

2023-01-17T15:22:15.993Z [MASTER] info: (SEARCH/POSTGRES) Rebuilding Index...

Then nothing happens... in Wiki.js I see a blue spinning circle in the upper right corner and after some time it runs into a request timeout (because of 60 seconds default timeouts in my ingress). I thought that maybe the request really takes longer and increased timouts up to 1 hour for testing, but it didn't help - the circle spins for 1 hour until the timeout appears.

Next I thought there could be something wrong with the PostgreSQL database instance. So I've deployed an Elasticsearch instance and tested with this one. Same result: it just runs into a timeout. I verified that Elasticsearch works by using it with a different Wiki.js instance.

Logs in the working instance look like this:

2023-01-17T15:29:48.540Z [MASTER] info: (SEARCH/ELASTICSEARCH) Initializing...
2023-01-17T15:29:48.557Z [MASTER] info: (SEARCH/ELASTICSEARCH) Initialization completed.
2023-01-17T15:30:05.707Z [MASTER] info: (SEARCH/ELASTICSEARCH) Rebuilding Index...
2023-01-17T15:30:05.791Z [MASTER] info: (SEARCH/ELASTICSEARCH) Creating index...
2023-01-17T15:30:06.427Z [MASTER] info: (SEARCH/ELASTICSEARCH) Sending batch of 12...
2023-01-17T15:30:07.055Z [MASTER] info: (SEARCH/ELASTICSEARCH) Index rebuilt successfully.

In the non working instance I just see this, afterwards it remains like this until the timeout appears:

2023-01-17T15:12:08.510Z [MASTER] info: (SEARCH/ELASTICSEARCH) Initializing...
2023-01-17T15:12:08.518Z [MASTER] info: (SEARCH/ELASTICSEARCH) Initialization completed.
2023-01-17T15:12:23.168Z [MASTER] info: (SEARCH/ELASTICSEARCH) Rebuilding Index...
2023-01-17T15:12:23.241Z [MASTER] info: (SEARCH/ELASTICSEARCH) Creating index...

The Wiki.js instance, where searches don't work, has about 150 pages. It doesn't seem to be that big and I guess it should be indexed in a few seconds, if everything works well.

Any ideas what could be wrong here? Or advices on what to change or test?
I'd be very happy if this starts working. :-)

[skyreacher]

I've done some more testing today and found out that there are more things, which do not work on that Wiki.js instance. For example storage to local file system or a git repository also does not work.

It looks like wiki.js cannot read or export contents from the database for some reason.

But when navigating through pages in the Wiki.js UI everything is normal, I can read, edit and create pages.

Any ideas what could be wrong with the database or what I could do to fix these problems?

[skyreacher]

I've done a lot more testing today and found something interesting. I did a DB backup and deleted pages in order to find out if a page causes problems. And in fact rebuild index started to work after I deleted some pages.

After several further tests I found out that it doesn't depend on a specific page but the number of pages. As soon as there are 100 or more pages in my Wiki.js instance, it stops working.

I modified the rebuild function in /wiki/server/modules/search/postgres/engine.js to limit selected pages to 99 -> with this restriction it started to work.

I guess this is not standard behaviour, or is it? Any ideas why there could be such a restriction and how I could make 100 or more pages work?

Here's the slightly modified rebuild function, that I used for testing:

  async rebuild() {
    WIKI.logger.info(`(SEARCH/POSTGRES) Rebuilding Index...`)
    await WIKI.models.knex('pagesVector').truncate()
    await WIKI.models.knex('pagesWords').truncate()
    await pipeline(
      WIKI.models.knex.column('path', 'localeCode', 'title', 'description', 'render').select().from('pages').where({
        isPublished: true,
        isPrivate: false
      }).limit(99).stream(),
      new stream.Transform({
        objectMode: true,
        transform: async (page, enc, cb) => {
          WIKI.logger.info(`handling page ${page.path} ...`)
          const content = WIKI.models.pages.cleanHTML(page.render)
          await WIKI.models.knex.raw(`
            INSERT INTO "pagesVector" (path, locale, title, description, "tokens", content) VALUES (
              ?, ?, ?, ?, (setweight(to_tsvector('${this.config.dictLanguage}', ?), 'A') || setweight(to_tsvector('${this.config.dictLanguage}', ?), 'B') || setweight(to_tsvector('${this.config.dictLanguage}', ?), 'C')), ?
            )
          `, [page.path, page.localeCode, page.title, page.description, page.title, page.description, content, content])
          cb()
        }
      })
    )

    await WIKI.models.knex.raw(`
      INSERT INTO "pagesWords" (word)
        SELECT word FROM ts_stat(
          'SELECT to_tsvector(''simple'', "title") || to_tsvector(''simple'', "description") || to_tsvector(''simple'', "content") FROM "pagesVector"'
        )
      `)

    WIKI.logger.info(`(SEARCH/POSTGRES) Index rebuilt successfully.`)
  }

[skyreacher]

Today I did a DB dump from Kubernetes and set Wiki.js and DB up locally. Everything worked fine.

Then I created a new PostgreSQL DB in Kubernetes and reimported the dump there. Then configured Wiki.js to use the new database.

Now everything works fine.

I have no idea what was wrong with the old DB, but there must have been something wrong (or configured differently) with it.

[btittelbach]

My ovservations with using Wiki.js 2.5.301 and elasticsearch as search provider are:

• Clicking „Rebuild Index“ deletes the index from elasticsearch, but does NOT rebuild it
• editing a page causes it to be re-indexd

the result being, that I can only search for pages that have been edited since „Rebuild Index“ has been clicked and nothing else