Skip to content

Latest commit

 

History

History
176 lines (131 loc) · 4.83 KB

README.md

File metadata and controls

176 lines (131 loc) · 4.83 KB
Raw

Ansible tooling for deploying the Transparency Toolkit

THIS IS PROBABLY NOT WORKING CORRECTLY

These recipes are being tested on Debian-stable and Debian-testing.

TODO pipeline diagram

DocUpload -> OCRServer -> IndexServer -> DocManager -> Catalyst -> LookingGlass w/ever

Ports used

Port Service Description
3000 DocManager Document storage
3001 LookingGlass Frontend
9002 Stanford-NER
9004 Catalyst
9292 DocUpload Small interface for uploading files.
9393 OCRServer
9494 IndexServer
9998 Tika Component of OCRServer.

Prerequisites

This repository contains a number of Ansible recipes to help deploy the Transparency Toolkit tools on servers.

The .yml files in the top directory (Catalyst.yml, DocUpload.yml, ...) are so-called "Ansible Playbooks."

To use our playbooks you must have the Ansible deployment tools installed, the sudo utility, and the Python bindings to libapt-pkg bindings. These can be installed with the following commands:

sudo apt update \
  && sudo apt install -y -q --no-install-recommends \
       ansible python-apt sudo

Each of the playbooks has a number of configuration options that you can specify using the --extra-vars argument, which takes a JSON dictionary.

We will be documenting the configuration options for each playbook below, but you can also manually consult the options for each role in the roles/*/defaults/main.yml files.

Hosted instance

See the server-scripts README for details on how to deploy a server that can automatically provision these services.

Installing DocUpload manually

ansible-playbook -v --ask-become-pass --forks 10 -c local DocUpload.yml \
  --extra-vars "{
    'ocrserver_url': 'http://127.1.2.3:9393',
    'lookingglass_url': 'https://demo.transparency.tools/',
  }"

Additional configuration options:

# The user under which to run the DocUpload service:
# (defaults to installing user)
docupload_user

# Directory into which data will be written before it is
# sent upstream to the OCRServer:
docupload_tmpdir

# The IP or hostname the DocUpload service will bind to
# (listen on, on port 9292):
docupload_ip: '127.0.0.1'

Installing IndexServer manually

ansible-playbook -v --ask-become-pass --forks 10 -c local IndexServer.yml \
  --extra-vars "{
    'docmanager_url': 'http://127.1.2.3:3000',
  }"

Additional configuration options:

# The user under which to run the DocUpload service:
# (defaults to installing user)
indexserver_user

docmanager_url

Installing DocManager manually

ansible-playbook -v --ask-become-pass --forks 10 -c local DocManager.yml \
  --extra-vars "{
    'docmanager_url': 'http://127.0.0.1:3000',
  }"

Additional configuration options:

docmanager_user
docmanager_ip

postgres_db
postgres_username

catalyst_url

Installing OCRServer

ansible-playbook -v --ask-become-pass --forks 10 -c local OCRServer.yml

Additional configuration options:

ocrserver_user

tika_version: '1.19.1'

indexserver_url: http://127.0.0.1:9494

SystemD services

Our services are installed as systemd services. Quick cheatsheet for managing systemd services:

Action to perform Shell command
View status sudo systemctl status docupload
Restart sudo systemctl restart docupload
View logs sudo journalctl -u docupload
Live view (tail) logs sudo journalctl -fu docupload

Environment variable overrides

You can override the default configuration options after installation by appending to files in /etc/systemd/MY.SERVICE.service.d/*, or by creating new files in those directories.

To override the default, append your line after # END ANSIBLE MANAGED BLOCK. The ansible scripts will update those sections with the upstream defaults, so custom changes have to be below those.

Example: /etc/systemd/system/docupload.service.d/gpg_signer.conf:

# BEGIN ANSIBLE MANAGED BLOCK
[Service]
Environment="gpg_signer='12345678'"
# END ANSIBLE MANAGED BLOCK
Environment="gpg_signer='my-real-keygrip-here'"

TODO NOTES