deploy: 17c7926

.buildinfo

@@ -0,0 +1,4 @@
+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: 6ea98c5a90654e6f280fe4cc28091e04
+tags: 645f666f9bcd5a90fca523b33c5a78b7

_sources/index.rst.txt

@@ -0,0 +1,45 @@
+Welcome to LmBISON - RIIS Analysis
+======================================
+
+The BISON repository contains data and scripts to annotate GBIF occurrence records
+with information regarding geographic location and USGS RIIS status of the record.
+
+
+Current
+------------
+
+.. toctree::
+    :maxdepth: 2
+
+    pages/about
+    pages/workflow
+
+Setup AWS
+------------
+
+.. toctree::
+    :maxdepth: 2
+
+    pages/aws/aws_setup
+
+Using BISON
+------------
+
+.. toctree::
+    :maxdepth: 2
+
+    pages/interaction/about
+
+History
+------------
+
+.. toctree::
+    :maxdepth: 2
+
+    pages/history/year4_planB
+    pages/history/year4_planA
+    pages/history/year3
+    pages/history/year5
+    pages/history/aws_experiments
+
+* :ref:`genindex`

_sources/pages/about.rst.txt

@@ -0,0 +1,12 @@
+About
+========
+
+The `Lifemapper BISON repository <https://github.com/lifemapper/bison>`_ is an open
+source project supported by USGS award G19AC00211.
+
+The aim of this repository is to provide a workflow for annotating and analyzing a
+large set of United States specimen occurrence records for the USGS BISON project.
+
+.. image:: ../.static/lm_logo.png
+  :width: 150
+  :alt: Lifemapper

_sources/pages/aws/automation.rst.txt

@@ -0,0 +1,9 @@
+Create lambda function to initiate processing
+------------------------------------------------
+
+* Check for existence of new GBIF data
+* Use a blueprint, python, "Get S3 Object"
+* Function name: bison_find_current_gbif_lambda
+* S3 trigger:
+
+    * Bucket: arn:aws:s3:::gbif-open-data-us-east-1

_sources/pages/aws/aws_setup.rst.txt

@@ -0,0 +1,109 @@
+AWS Resource Setup
+********************
+
+Create policies and roles
+===========================================================
+
+The :ref:`_bison_ec2_s3_role` allows an EC2 instance to access the public S3 data and
+the bison S3 bucket.  Its trust relationship grants AssumeRole to ec2 and s3 services.
+This role will be assigned to an EC2 instance that will initiate
+computations and compute matrices.
+
+The :ref:`_bison_redshift_s3_role` allows Redshift to access public S3 data and
+the bison S3 bucket, and allows Redshift to perform glue functions. Its trust
+relationship grants AssumeRole to redshift service.
+
+Redshift Namespace and Workgroup
+===========================================================
+
+A namespace is storage-related, with database objects and users.  A workspace is
+a collection of compute resources such as security groups and other properties and
+limitations.
+https://docs.aws.amazon.com/redshift/latest/mgmt/serverless-workgroup-namespace.html
+
+
+EC2 instance creation
+===========================================================
+
+Create (Console)
+--------------------------------
+* Future - create and save an AMI or template for consistent reproduction
+* via Console, without launch template:
+
+  * Ubuntu Server 24.04 LTS, SSD Volume Type (free tier eligible), Arm architecture
+  * Instance type t4g.micro (1gb RAM, 2 vCPU)
+  * Security Group: launch-wizard-1
+  * 15 Gb General Purpose SSD (gp3)
+  * Modify `IAM instance profile` - to role created for s3 access (bison_ec2_s3_role)
+  * Use the security group created for this region (currently launch-wizard-1)
+  * Assign your key pair to this instance
+
+    * If you do not have a keypair, create one for SSH access (tied to region) on initial
+      EC2 launch
+    * One chance only: Download the private key (.pem file for Linux and OSX) to local
+      machine
+    * Set file permissions to 400
+
+  * Launch
+  * Test by SSH-ing to the instance with the Public IPv4 DNS address, with efault user
+    (for ubuntu instance) `ubuntu`::
+
+    ssh  -i .ssh/<aws_keyname>.pem  ubuntu@<ec2-xxx-xxx-xxx-xxx.compute-x.amazonaws.com>
+
+
+Install software on EC2
+===========================================================
+
+Baseline
+------------
+* update apt
+* install apache for getting/managing certificates
+* install certbot for Let's Encrypt certificates
+* install docker for BISON deployment::
+
+    sudo apt update
+    sudo apt install apache2 certbot plocate unzip
+    sudo apt install docker.io
+    sudo apt install docker-compose-v2
+
+AWS Client tools
+--------------------
+
+* Use instructions to install the awscli package (Linux):
+  https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html.
+* Make sure to use the instructions with the right architecture (x86 vs Arm)
+* Test by listing the contents of bison bucket (permission from role bison_ec2_s3_role)::
+
+  aws s3 ls s3://bison-321942852011-us-east-1/input/
+
+SSL certificates
+------------------
+
+* Create an SSL certificate on the EC2 instance.
+* For testing/development, use self-signed certificates because Cerbot will not create
+  certificates for an AWS EC2 Public IPv4 DNS, or an IP address.
+
+  * Edit the docker-compose.yml file under `nginx` service (which intercepts all web
+    requests) in `volumes` to bind-mount the directory containing self-signed
+    certificates to /etc/letsencrypt::
+
+    services:
+    ...
+      nginx:
+      ...
+      volumes:
+        - "/home/ubuntu/certificates:/etc/letsencrypt:ro"
+
+BISON code
+---------------------
+
+* Download the BISON code repository::
+
+  git clone https://github.com/lifemapper/bison.git
+
+* Edit the .env.conf (Docker environment variables) and nginx.conf (webserver address)
+  files with the FQDN of the server being deployed. For development/testing EC2 servers,
+  use the Public IPv4 DNS for the EC2 instance.
+
+Launch BISON docker instances
+-----------------------------------

_sources/pages/aws/roles.rst.txt

@@ -0,0 +1,39 @@
+Roles, Policies, Trust Relationships
+=========================================
+
+.. _bison_redshift_s3_role:
+
+bison_redshift_s3_role
+------------------------------
+
+* Trusted entity type: AWS Service
+* for Redshift - Customizable
+
+  * TODO: change to Redshift - Scheduler when automated
+
+* Policies:
+
+  * bison_s3_policy (read public/GBIF S3 data and read/write bison S3 data)
+  * redshift_glue_policy.json (for Redshift interactions)
+  * AmazonRedshiftAllCommandsFullAccess (AWS managed)
+  * AmazonS3FullAccess (AWS managed)
+
+* Trust policy:
+
+  *
+
+.. _bison_ec2_s3_role:
+
+bison_ec2_s3_role
+------------------------------
+
+* Trusted entity type: AWS Service
+* for S3
+* Includes policies:
+
+  * bison_s3_policy.json (read public/GBIF S3 data and read/write bison S3 data)
+  * SecretsManagerReadWrite (AWS managed)
+
+* Trust relationship:
+
+  * ec2_s3_role_trust_policy.json edit trust policy for both ec2 and s3