Automated Integration Testing on Multiple Environments

[loreanvictor]

Since most new features are added (and tested) in one singular environment (typically macOS), they always have a chance of failing in other environments. For example this is the latest such bug, which could have easily been prevented if there was automated testing in multiple environments.

Testing the CLI is perhaps a bit more involved than typical unit testing since most of its functionality can only be properly tested when its interactions with CODEDOC and standard CODEDOC repositories are taken into consideration. More importantly, the testing needs to be done in multiple environments (macOS, Linux, and Windows), specifically as differences between Windows and the rest causes lots of easily preventable problems (though we have also had cases of Linux being the exceptional case). Since I haven't personally undertaken such an automated testing mechanism before, I suspect some research would also be needed to assess what technologies and options are available. Due to this complexity I suspect such tests would also further complicate setting up a dev environment for the project, which is an unavoidable result.

So what we need is:

• There is a convenient enough way of expressing desired behavior of the CLI in form of tests.
  • Ideally these descriptions are pretty easy to read and understand
  • Ideally the multi-environment nature of tests are abstracted away from the tests themselves
• Each push to master is tested
  • Ideally tests can be conducted locally as well before a push, though that might not be possible/easy
  • Ideally integrated with GitHub's checks somehow (so there is an email / notification when tests fail)
• Tests are conducted in at least 3 different environments (macOS, Windows, Linux)
  • Ideally we generally avoid mocking stuff for these tests, since most of the bugs we've had so far come from actual differences in behavior of different environments, which would be masked away by mocking.

Note that this is not unit testing. While having unit tests would also be really useful, so far we haven't had any situation where the project would benefit from unit-tests (though that definitely changes as the project grows and more people start to contribute).