Given a success message that results pass but nothing is posted

[jbhennin]

Receive: Publishing 2 test result(s) to 'mytestrail.net'
Log into testrail and no results have been logged
I am using a user and apikey

I suspect that there is an issue with the post request you guys are submitting as I have used dummy url and bad username and no errors are logged to the console.

[kolodiy]

@jbhennin,
Having the same issue when using reporter with protractor, and so far no issues when using reporter for integration tests.
Receiving:
1 passing (8s)

Publishing 1 test result(s) to https://test_domain/index.php
And no results are published.

Here is the Protractor config:
exports.config = {
framework: 'mocha',
mochaOpts: {
recursive: true,
reporter: 'mocha-testrail-reporter',
reporterOptions: {
domain: 'test_domain',
username: 'test_username',
password: 'test_password',
projectId: projectID,
suiteId: suiteID,
},
fullTrace: true,
noExit: true,
timeout: 9999
},

Seems like it is configuration issue, @awaragi can you suggest what I can do to make it work with Protractor?

Thanks,
Roman.

[awaragi]

I will take a look at the issue. I have tested the reporter primary with mocha vanilla so I don't have a projet with protractor. If this is a small test projet, can you please share it so that I can look deeper?

[kolodiy]

@awaragi ,
I created trial account on testRail, and simple test with protractor example, so you may look deeper. Here it is:
https://github.com/kolodiy/protractorTestRailReporterBug
All mocha options are stored in local_testRail_conf.js

Prerequisites:
Use npm to install Protractor globally with:

npm install -g protractor This will install two command line tools, protractor and webdriver-manager. Try running protractor --version to make sure it's working.

The webdriver-manager is a helper tool to easily get an instance of a Selenium Server running. Use it to download the necessary binaries with:

webdriver-manager update Now start up a server with:

webdriver-manager start
To run the project:

npm install
protractor test/local_testrail_conf.js

Thanks,
Roman

[jbhennin]

@awaragi
Here is a simple test that shows the issue I am having:
https://github.com/jbhennin/tests

npm run exampleRequestTR

[awaragi]

okay. so I've looked at the issue and the problem is that protactor is not waiting for the submissions to be processed. Basically the submission of results to testrail is getting lost!

In Mocha, the --no-exit flag was enough to tell node not to quit until all events have been processed. protactor does not seems to be passing this flag to mocha runner.

I will try to see if I can rework the code to use a different http agent.

[awaragi]

the noExit configuration value you added to the mochaOptions is only used in the bin/mocha script therefore it is not applicable to the mocha runner via code as called by protactor. This is harder than I thought as the concept of sync http request does not exists natively in nodejs without external modules.

(see mochajs/mocha@55a33ce)

[awaragi]

@jbhennin you simply need to add --no-exit to your mocha call for the publication to work.

[kolodiy]

@awaragi , so is there any chance to see it working with Protractor in the nearest future?

[awaragi]

Hi,
I tried to replace unitrest with sync-request but for some kind of reason it is not working for me. I keep getting ECONNRESET error even when trying simple request to https://google.com. If you want to take a crack at it. the function to change is _post in src/lib/testrail.ts

Here is the code which fails on the = request(....

    private _post(api: String, body: any, callback: Function, errorCallback?: Function) {
        try {
            let qs = `/api/v2/${api}`;
            let authorization = "Basic " + new Buffer(this.options.username + ":" + this.options.password).toString("base64");
            var response = request("POST", this.base, {
                headers: {
                    'content-type': 'application/json',
                    Authorization: authorization,
                },
                qs: qs,
                body: JSON.stringify(body)
            });

            if (response.statusCode >= 300) {
                console.error("Error: %s", response.body);
                if (errorCallback) {
                    errorCallback(response.body);
                }
            }
            callback(response.body);
        } catch (error) {
            console.error("Error", error);
            if (errorCallback) {
                errorCallback(error);
            }
        }


    }

[howieweiner]

I just had the same issue using Mocha. I've resolved it two ways - Firstly, I added the --no-exit arg. This informed me that there was an authentication problem. Seems that authentication fails when using the API Key. Works fine with password.

[awaragi]

That's exactly my proposed solution for mocha. Unfortunately protractor does not have the same flag.

As for the API key, I have it linked to a production environment and it works great. I am surprised it does not work for you!

[howieweiner]

You're right. The API key does work. My bad. Didn't realise you need to save the settings page in Testrail after generating the key.

[camlegleiter]

Looks like this is related to mochajs/mocha#2541. It looks like the mocha Runner doesn't really support asynchronous code well within its EventEmitter functionality.

We ended up using --no-exit and manually calling process.exit(failureCount) after our asynchronous code finished running. Not ideal by any means, but it does let us use this reporter in CI.

[awaragi]

Unfortunately moch reports and jasmine are not the same. So I don’t think so.

[michaelseno]

Hi @awaragi I tried using this plugin for the mocha test that I have. and same concern is that it will say success but nothing is published in testrail. I did try to put --no-exit flag but it seems to have an error Error: undefined. I tried puting it in mocha.opts or in package.json run tag.

[awaragi]

Hi, do you mind sharing a little bit of your project? got quite few projects internally that use it and it seems to work. Ensure that the project in testrail is multi-suite otherwise it will not work.

[michaelseno]

Hi, I'm sorry but can you elaborate more on the multi-suite part? i'm new with testrail. this is the script / test i'm running.

describe('Test Web', function () {
  let webClient;
  let csvData;
  let loginPage;

  before(async function () {
    this.timeout(config.waitTime);
    webClient = deviceMgmt.startDevice(device.browser_chrome);
    csvData = await csvParser.getCSVData('1', 'credentials');
    await webClient.init();
    loginPage = new LoginPage(webClient);
  });

  after(async function () {
    this.timeout(config.waitTime);
    const logs = webClient.log('browser');
    await nativeLogs('Browser', logs, 'login');
    await webClient.session('delete');
  });

  it('C576544 Open Browser URL', async function () {
    this.timeout(config.waitTime);
    await loginPage.openLoginPage(csvData.url);
  });

  it('C576545 Enter Username and Password', async function () {
    this.timeout(config.waitTime);
    await loginPage.enterUsername(csvData.username);
    await loginPage.enterPassword(csvData.password);
  });

  it('C576546 Click Login', async function () {
    this.timeout(config.waitTime);
    await loginPage.clickOnLogin();
  });
});

by the way. do we need to manually create a test run for this? or it will automatically create based on the project and suite you provide in the opts file? since I have 5 Test Suites in the project in testrail containing test cases. but I only want to test 1 Test Suite? is that possible?

[michaelseno]

I was able to make it published. I think the reason why its not publishing because of the unhandled rejection on self certificate. And I was able to make it work by putting this into the script. process.env.NODE_TLS_REJECT_UNAUTHORIZED = '0';

[awaragi]

Most likely that was the problem. Ive only tested it with hosted instance of Testrail. The reporter accepts several parameters such as project id and suite id, so you should be able to publish tests for 1 suite without any issues. The opposite is not supported (and does not make senses since test runs are based on one suite at a time).

[michaelseno]

Thank you for the clarification.

[anjalibangoria]

I am able to solve problem with use of promises.
For example,
on('after:spec', (spec, results) => {
return new Promise((resolve, reject) => {
resolve(results);
})
});