Merge pull request #2667 from carpentries/feature/2462-emails-documen…

…tation

[Emails] Updated technical documentation

docs/design/application_design.md

@@ -18,10 +18,11 @@ very difficult when it came to move the underlying model to another app.
 The new application structure contains:
 
 * `api` - for CRUD API interface provided by [DRF](https://www.django-rest-framework.org/),
-* `autoemails` - for automated emails application,
+* `autoemails` - for automated emails application (v1, 2019) - now being deprecated,
 * `communityroles` - for community roles project application,
 * `consents` - for consents project application,
 * `dashboard` - for admin and instructor dashboard views,
+* `emails` - for current automated emails application (v2, 2023),
 * `extcomments` - for overriding some functions from
   [`django-contrib-comments`](https://django-contrib-comments.readthedocs.io/en/latest/quickstart.html)
   third party application for comments,

docs/design/database_models.md

@@ -209,7 +209,7 @@ Represents a requirement that a prospect future instructor need to pass.
 This model also falls into `trainings` domain.
 
 ### `TrainingProgress`
-Intermediate table for M2M between `Person`, `TrainingRequirement`, and optionally 
+Intermediate table for M2M between `Person`, `TrainingRequirement`, and optionally
 `Involvement`. Indicates "pass",
 "fail", or "asked to repeat" progress of a person over a particular requirement.
 Once all required requirements are passed, person can become an instructor.
@@ -335,7 +335,7 @@ events, it works for The Carpentries' community.
 
 ----------------------------------------------------------------------------------------
 
-## Automated Emails application - `autoemails/models.py`
+## Automated Emails (v1, 2019) application - `autoemails/models.py`
 
 ### `EmailTemplate`
 Represents email template to be used by a triggered email action.
@@ -354,5 +354,22 @@ and links to Redis entry maintained by RQ library.
 
 ### `Involvement`
 
-Represents an activity that can be completed as part of the Get Involved step of 
+Represents an activity that can be completed as part of the Get Involved step of
 Instructor Training checkout. Referenced by [`TrainingProgress`](#trainingprogress).
+
+----------------------------------------------------------------------------------------
+
+## Emails (v2, 2023) application - `emails/models.py`
+
+### `EmailTemplate`
+Represents email template to be used by a scheduled email.
+
+### `ScheduledEmail`
+Model containing details of a scheduled email. This is used for queueing emails to be
+sent, as well as capturing any failed, locked, or cancelled emails.
+When accessed from AMY API, it can return a list of emails that should be attempted to
+be sent by the email worker.
+
+### `ScheduledEmailLog`
+Represents a log entry for a scheduled email. This is used to track the status changes
+of an email.

docs/design/projects/2019_automated_emails.md

@@ -1,4 +1,4 @@
-# Automated emails
+# Automated emails v1 (2019)
 
 GitHub project link unavailable.
 

docs/design/projects/2023_automated_emails.md

@@ -0,0 +1,144 @@
+# Automated emails v2 (2023)
+
+Github project: https://github.com/orgs/carpentries/projects/10
+
+The aim of this project is to replace the current system of automated emails, which is
+based on Redis and Python RQ, with a new system based on decoupled systems.
+The new system should be more robust, easier to maintain, and allow for more complex
+scheduling of emails, at the same time allowing to add the new types of emails more
+easily.
+
+Apart from being decoupled, another important feature of this system is that it fetches
+the newest representation of objects used in the scheduled email content using AMY API,
+and creating new actions is simpler and more elegant thanks to the use of
+[Django signals](https://docs.djangoproject.com/en/5.0/topics/signals/).
+
+
+## Infrastructure
+
+At the core of automated emails there are:
+
+* AMY API exposing endpoints for managing scheduled emails,
+* AMY API exposing endpoints for details of objects used in emails' context,
+* A separate application for accessing the AMY API and sending emails.
+
+At the core of the new system in AMY there are used:
+
+* Django signals for triggering actions,
+* a controller for managing scheduled emails (scheduling, rescheduling, updating,
+  cancelling, etc.),
+* types defined for improved type-safety,
+* management panel for viewing and managing scheduled emails.
+
+## How it works
+
+Emails can be scheduled by sending a signal with a specific
+payload. There always is one signal for scheduling the email, but
+some emails allow for updating or cancelling them, which is done
+through two other signals.
+
+To help deciding if an email should be sent, updated or cancelled,
+these emails provide strategies, which implement checks for
+conditions that should be met for the email to be sent (or updated,
+or cancelled).
+
+Once the email has been scheduled, it is stored in the database as
+`ScheduledEmail` record. This record contains all the information
+needed to send the email, including the email's content, the time
+when it should be sent, and the email's context objects.
+
+The context contains information about the objects that should be
+used when generating MD and HTML content of the email. It consists of
+model name and model's primary key, which allows to fetch the object
+from the API when rendering the email.
+
+Once emails have been scheduled, they can be retrieved by the email worker. This is
+a Python lambda application that runs every 5 minutes, fetches the emails and sends
+them. The accurate algorithm is described in another section.
+
+## Email worker algorithm
+
+After the worker sets up, it fetches all emails that should be sent by now. Then it
+processes them individually asynchronously:
+
+1. Lock the email record to prevent UI work on it.
+2. Create context objects for email recipient list and email body content.
+3. Create the email context with actual data from the API.
+4. Create the email recipient list with actual data from the API.
+5. Render the email body and subject with Jinja2 and the context.
+6. Render the Markdown version of the email body (this generates the HTML).
+7. Send the email.
+8. Update the email record with the status.
+
+At any step this process can fail and the email will be marked as failed. The worker
+will pick it up again in the next run.
+
+## Implementation of new actions
+
+All actions are defined in `emails.actions` module. Each action is a class
+inheriting from `BaseAction` class (for scheduling emails). If a specific action
+could allow for updating or cancelling, it should consist of 2 additional classes
+inheriting from `BaseActionUpdate` and `BaseActionCancel` respectively.
+
+Each action must implement the following required methods and fields:
+
+1. inheriting from `BaseAction`:
+    * `signal` - parameter that contains a value uniquely identifying the action signal,
+      and therefore also the email template
+    * `get_scheduled_at()` - method that calculates when the action should be run
+    * `get_context()` - method that returns the context for the email
+    * `get_context_json()` - method that returns the context for the email in JSON
+      format (this is used by the email worker to fetch the context from the API)
+    * `get_generic_relation_object()` - method that returns the main object for the
+      email (e.g. an event or a person)
+    * `get_recipients()` - method that returns the list of recipients of the email
+    * `get_recipients_context_json()` - method that returns the recipients of the email
+      in JSON format (this is used by the email worker to fetch the recipients from the
+      API)
+
+2. inheriting from `BaseActionUpdate`:
+    * the same fields and methods as in `BaseAction` class
+
+3. inheriting from `BaseActionCancel`:
+    * the same fields and methods as in `BaseAction` class, except for `get_recipients()`
+      and `get_scheduled_at()` methods, which are not needed for the cancelling action.
+
+Each base class implements a `__call__()` method that in turn uses appropriate
+`EmailController` method to schedule, update, or cancel the email.
+
+### Implementing a new action - checklist
+
+1. Add new action signal name to `emails.signals.SignalNameEnum` enum.
+2. Define the context TypedDict in `emails.types` module. This should be a dictionary
+   with keys and types of values that will be passed to the email template as context.
+3. Define the kwargs TypedDict in `emails.types` module. This should be a dictionary
+   with keys and types of values that will be passed to the action's constructor (when
+   the signal for email is being sent).
+4. Define the action class in a new module in `emails.actions` package. This class should
+   inherit from `BaseAction` class and implement all required methods.
+5. If the action should allow for updating or cancelling, define additional classes
+   inheriting from `BaseActionUpdate` and `BaseActionCancel` respectively.
+6. Create receivers as instances of the action classes. Link the receivers to the
+   appropriate signals in `emails.signals` module:
+   ```python
+   receiver = MyAction()
+   signal.connect(receiver)
+   ```
+
+7. If the action consists of scheduling, updating, and cancelling, create `action_strategy` and `run_action_strategy` functions. Follow examples from other actions.
+
+
+### Using a new action
+
+If the action contains a strategy, then using it is quite simple:
+
+```python
+run_action_strategy(
+    action_strategy(object),
+    request,
+    object,
+)
+```
+
+Strategies may accept other parameters, but the selected strategy and request (as in
+Django view request) are required.

docs/design/projects/README.md

@@ -5,11 +5,11 @@ Documentation provided for each of the functional projects.
 [GitHub link to projects](https://github.com/carpentries/amy/projects?type=classic).
 
 
-* [Automated emails](./2019_automated_emails.md)
+* [Automated emails v1 (2019)](./2019_automated_emails.md)
 * [Memberships](./2021_memberships.md)
 * [Consents](./2021_consents.md)
 * [Profile Archival](./2021_profile_archival.md)
 * [Community Roles](./2021_community_roles.md)
 * [Instructor Selection](./2021_instructor_selection.md)
 * [Single Instructor Badge](./2021_single_instructor_badge.md)
-
+* [Automated emails v2 (2023)](./2023_automated_emails.md)

docs/design/server_infrastructure.md

@@ -187,6 +187,11 @@ automated emails feature.
 > **Note March 2023:** this was not achieved, and is required for migrating production
 > to containerized (cloud native) solution.
 
+> **Note June 2024:** this is part of a work related to Emails project (v2, 2023),
+> scheduled to be completed by the end of 2024. Production will definitely be de-coupled
+> from RQ worker and scheduler, but it's not clear yet when will the production be
+> containerized.
+
 ### Containerize the application to make it immutable
 
 AMY should be containerized. It's already possible to build a Docker image with AMY,