Skip to content

Simple Node.js script based on producer-consumer problem. For transfering documents between two elasticsearch servers or between indices of the same elasticsearch server.

Notifications You must be signed in to change notification settings

gaurang847/elasticsearch-document-transfer

Repository files navigation

elasticsearch-document-transfer

npm version

Simple Node.js script to transfer documents between two elasticsearch servers or between indices of the same elasticsearch server.

Implementation based on consumer-producer problem.
Since Node.js is single-threaded, producer and consumer cannot simultaneously access the buffer. And hence, as long as the asynchronous nature of Node.js is handled properly, there shouldn't be any issues related to concurrency.

Usage:

  1. Clone the repo.
  2. Run npm install in the directory.
  3. Add config.js using config-sample.js.
    Exercise caution while setting source and target elastic hosts (and while editing this file in general).
  4. (Optional) Set appropriate values in options.js
  5. Run the script
    node index.js
    

Note:

If options.consume.byFile is set to true, the documents will be written to a file in a format as expected by the bulk API of elasticsearch. The path of file must be specified in config.js.

If options.consume.byElastic is set to true, the documents will be directly added to the target elastic server. The connection details should be present in config.js.

About

Simple Node.js script based on producer-consumer problem. For transfering documents between two elasticsearch servers or between indices of the same elasticsearch server.

Topics

Resources

Stars

Watchers

Forks

Packages

No packages published