Readme for the Testing - detailed instructions for functional testing usage and addition of new tests

[nanda-katikaneni]

🗣 Description

Create a high-level Readme for the testing folder with detailed instructions for:

• Functional Testing - Structure and High Level Details
• Functional Testing Usage with Examples
• Functional Testing - Adding New Test Cases with Examples

💭 Motivation and context

closes #735

🧪 Testing

• Run functional tests for a product using the instructions
• Add a new test using the instructions

✅ Pre-approval checklist

• This PR has an informative and human-readable title.
• PR targets the correct parent branch (e.g., main or release-name) for merge.
• Changes are limited to a single goal - eschew scope creep!
• Changes are sized such that they do not touch excessive number of files.
• All future TODOs are captured in issues, which are referenced in code comments.
• These code changes follow the ScubaGear content style guide.
• Related issues these changes resolve are linked preferably via closing keywords.
• All relevant type-of-change labels added.
• All relevant project fields are set.
• All relevant repo and/or project documentation updated to reflect these changes.
• All automated checks (e.g., linting, static analysis, unit/smoke tests) passed.

✅ Pre-merge checklist

• PR passed smoke test check.

• Feature branch has been rebased against changes from parent branch, as needed

  Use Rebase branch button below or use this reference to rebase from the command line.

• Resolved all merge conflicts on branch

• Notified merge coordinator that PR is ready for merge via comment mention

✅ Post-merge checklist

• Feature branch deleted after merge to clean up repository.
• Verified that all checks pass on parent branch (e.g., main or release-name) after merge.

[schrolla]

Recommend making the filename all caps to be consistent with the convention in the main directory for documentation files. Not a deal breaker, just a suggestion.

[james-garriss]

Recommend putting all documentation (other than the base readme) in one folder (with subfolders as necessary), perhaps called docs or documentation.

[rgbrow1949]

The instructions look good, but this branch needs a rebase and there are unit tests that are currently failing

[tkol2022]

Pushed to halibut so that we have more time to make content revisions and complete a review.

[tkol2022]

In the instructions where Pester is installed and you have prerequisites, you are missing a step to install Selenium. That is before running the Update Selenium script.
Install-Module Selenium

[tkol2022]

I noticed in the section where we tell the reader to "kill any ChromeDriver.exe processes", I found that those processes are sometimes hidden under the Powershell app and you cannot locate them in the main process list. Although the ChromeDriver.exe is not shown in my screenshot below, that is where I saw an instance of it earlier. Add this information to the instructions.

[tkol2022]

Today when I tried running the functional tests, I was a getting a random error (forgot to take a screenshot) that was unrelated to any of our code. Selenium just wasn't working right for whatever reason and I cannot recall this ever happening before. I updated the ChromeDriver.exe but that didn't fix the problem so I ended up having to do what's below and now I can run the orchestrator again:

uninstall-module Selenium
install-module Selenium
./Testing/Functional/SmokeTest/UpdateSelenium.ps1

[tkol2022]

The README created for this PR is oriented to users (i.e. developers, testers) that want to execute the test orchestrator. I think you should add a couple of sentences at the top that makes this more obvious and change the title of the document to ScubaGear Functional Testing Automation for Developers and Testers.

On that note, I had developed a couple of ancillary guides for us admins which are related to setting up the infrastructure (service principals, user provisioning, etc.). Can we add a link to those GitHub issues in this document for easy reference?

• How to setup the permissions required to execute the automated functional test orchestrator How to setup the permissions required to execute the automated functional test orchestrator #589

• How to setup a tenant with the necessary AAD conditional access policies to run the Automated Functional Test Orchestrator How to setup a tenant with the necessary AAD conditional access policies to run the Automated Functional Test Orchestrator #591

[tkol2022]

In your Functional Testing Usage section make it clear that the addition of the Thumbprint field to the -Data parameter is what makes the orchestrator run in service principal mode and that the thumbprint must correspond to the end user's client certificate which should be installed on the machine running the test plans.

See #592 for example.

I would just copy the following sections verbatim because they contain a lot of details and some of that stuff is missing from your README.
#592 (comment)
#592 (comment)

[tkol2022]

Throughout the document I think it would be better to use the voice "you need to do this" instead of the third person "the user should do this". This makes the document easier to follow as a set of instructions directed at the reader.

[tkol2022]

Add a section with the Powershell code to generate a user client certificate.

$CertName = "ScubaServicePrincipal"
$CertParams = @{
    Subject = $CertName
    KeySpec = "KeyExchange"
    CertStoreLocation = "Cert:\CurrentUser\My"
}

$MyCert = New-SelfSignedCertificate @CertParams
Write-Output $MyCert


########## Exporting the cert public key (this is so you can upload the file to the Azure AD application as a credential)
Export-Certificate -Cert $MyCert -Type CERT -FilePath .\ScubaServicePrincipal.cer

[tkol2022]

Replace "AAD functional testing in service principal mode by filtering specific test cases" with "When developing or if you are testing pull requests you may need to execute just a single baseline policy in the respective YAML test plan file. To do that you will add Pester filter tags to the PesterConfig parameter. See an example to execute the policy MS.AAD.2.1v1 test cases below."

[tkol2022]

"Product functional tests are being run for multiple product-tenant combinations. Development team should check the successful completion of these runs for any regressions. "

It is not clear when the development team should check this? Is this more the job of the operations team?

Also, for performing regression tests, one option that a developer can exercise is to run the nightly product functional test via the GitHub actions against their branch although since that runs all of the products and all of the test cases, I think we should determine a more efficient way to do that if we ever want this option.

[tkol2022]

I don't see a reference to the sample functional test execution script that is reference in this comment. That might make it easier for developers.

#592 (comment)

[tkol2022]

In the Adding New Functional Tests section, add a checkbox for "Ensure all Not Applicable test cases for the affected policy are tested."

Both Sharepoint and AAD now include N/A policy checks. These are for instances where a policy is not checked for a specific reason. For example if AAD 3.1 is implemented, policy check 3.3 becomes Not Applicable and ScubaGear doesn't check it, but displays an N/A in the report.

[tkol2022]

New Functional tests section. "Ensure that any changes to HTML report output are tested." Not sure what the developer is supposed to do here and how those changes would be tested.

Also, for "Validate that all new functional tests pass on CI for the branch before creating the PR" not sure what CI stands for.

[tkol2022]

"compliant" is misspelled in numerous instances as "complaint".

[tkol2022]

I have completed my review and left revisions to be addressed.

[james-garriss]

When reviewing this PR, please note that there are 2 branches that begin with "735". The correct branch to review is this one:
https://github.com/cisagov/ScubaGear/tree/735-FunctionalTests-Documentation

[james-garriss]

Lots of good info in here; thanks for putting this together.

[james-garriss]

Suggested change

	This document outlines the ScubaGear software test automation and its usage. The document also contains instructions for adding new functional tests to existing automation suite.
	This document outlines the ScubaGear software test automation and its usage. The document also contains instructions for adding new functional tests to the existing automation suite.

[james-garriss]

Suggested change

	Running ScubaGear functional test automation require Windows compute or VM end system. This development system should be setup with Pester, Selenium, Chrome and PowerShell 5.1. The repository provides a few utility scripts to install and update these prerequisite components.
	Running the ScubaGear functional test automation requires a Windows computer or VM. This development system should be setup with Pester, Selenium, Chrome and PowerShell 5.1. The repository provides a few utility scripts to install and update these prerequisite components.

[james-garriss]

Suggested change

	Functional testing needs to be run on a Windows system with Chrome browser installed. Automation suite uses Selenium tool which requires Chrome driver for its web component testing.
	Functional testing needs to be run on a Windows system with the Chrome browser installed. The automation suite uses Selenium, which requires the Chrome driver for its web component testing.

[james-garriss]

Suggested change

	On your Windows development system, install an updated Pester module by running following command as an administrator:
	On your Windows development system, install an updated Pester module by running the following command as an administrator:

[james-garriss]

Suggested change

	Product functional tests are being run for multiple product-tenant combinations. Development team should check the successful completion of these runs for any regressions. Currently the following "product / tenant / other variant" combinations are run by a cron job:
	A GitHub workflow is used to execute multiple product functional tests every night, and they are being run for multiple product-tenant combinations. Development teams should check for the successful completion of these runs each morning to ensure that there are not any regressions. As of May 2024, the following "product / tenant / other variant" combinations are run each night:

[james-garriss]

Suggested change

	The current list of tests is found in a Slack canvas.

[james-garriss]

Suggested change

	```
	```powershell

[james-garriss]

Suggested change

	Sometimes the UpdateSelenium script has a problem getting rid of an older chromedriver.exe even if you think you have killed any running processes named chromedriver.exe. You might receive the error "Cannot remove item. Access to the path ...\chromedriver.exe is denied" as in the screenshot above. In that case you should follow the steps below:
	Sometimes the UpdateSelenium script has a problem getting rid of an older chromedriver.exe, even if you think you have killed any running processes named `chromedriver.exe`. You might receive the error `"Cannot remove item. Access to the path ...\chromedriver.exe is denied"`, as in the screenshot above. In that case, you should follow the steps below:

[james-garriss]

Suggested change

	- Open the path shown in the error message in Explorer (e.g. C:\Users\tkolovos\Documents\WindowsPowerShell\Modules\Selenium\3.0.1\assemblies)
	- Open the path shown in the error message in Explorer (e.g. `C:\Users\tkolovos\Documents\WindowsPowerShell\Modules\Selenium\3.0.1\assemblies`)

[james-garriss]

When I ran this command, I got the following warning:

WARNING: The version '5.5.0' of module 'Pester' is currently in use. Retry the operation after closing the
applications.

Please add this warning to the instructions and include instructions for how to respond to it.

[james-garriss]

Please add this error to the README and provide a response for resolving it:

Move-Item : Access to the path is denied.
At C:\Users\jgarriss\Documents\ScubaGear\Testing\Functional\SmokeTest\UpdateSelenium.ps1:90 char:9
+         Move-Item "$DriverTempPath\chromedriver-win64\chromedriver.ex ...
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : PermissionDenied: (C:\Users\jgarri...hromedriver.exe:FileInfo) [Move-Item], Unauthorized
   AccessException
    + FullyQualifiedErrorId : MoveFileInfoItemUnauthorizedAccessError,Microsoft.PowerShell.Commands.MoveItemCommand

[nanda-katikaneni]

In the instructions where Pester is installed and you have prerequisites, you are missing a step to install Selenium. That is before running the Update Selenium script. Install-Module Selenium

Added