GitBeaker Request Logging / Tracing?

[vallamost]

Description

It's unclear how I can track the number of requests made to GitLab from the gitbeaker NodeJS client or how I can see the HTTP requests & responses GitBeaker is getting. Is there some documentation somewhere that would let me enable and see request tracing and logging?

How can I enable HTTP request tracing to see all of the requests GitBeaker is making?
E.g. If I make a request to get all successful pipelines for a projectID, I want to see how many HTTP requests are made to complete that request and what the details of those HTTP requests are, latency, response codes, size, etc.

Checklist

• I have checked that this is not a duplicate issue.
• I have read the documentation.

[vallamost]

The block of code below was something I found that helps trace outgoing requests made by GitBeaker however it doesn't show the raw responses. You can put this in your main entry point file, e.g. main.js or app.js and you should see gitlab requests in your console log when you make them.

function enableRequestLogger() {
    const https = require('https');
    const original = https.request
    https.request = function(options, callback){
      console.log(options);
      console.log(`${new Date().toISOString()} ${options.method} ${options.protocol}//${options.hostname}${options.path}`);
      return original(options, callback);
    }
}

enableRequestLogger();

[jdalrymple]

Related to #764

[jdalrymple]

Full response object is include in the error when it occurs

[marcelstoer]

Full response object is include in the error when it occurs

That's not too helpful in many cases. We really need to see request including all headers and full response.

In case of e.g. HTTP 404 the response just says "not found" - with no information about where it was looking.

[jdalrymple]

So the full response is included, but the full request isnt. easy fix. Ill through up a PR in a bit

[marcelstoer]

I just realized I can actually do NODE_DEBUG=http,http2 node index.js. Dumps loads of information but it's not very concise and hard to read.

[trevor-vaughan]

Just had a case where curl works but @gitbeaker doesn't. This would definitely be super helpful!

[jdalrymple]

Hoping to merge the latest stuff over the weekend! Keep an eye out.

[jdalrymple]

Just had a case where curl works but @gitbeaker doesn't. This would definitely be super helpful!

Also, just out of curiosity, did it have to do with a headers timeout?

[trevor-vaughan]

It didn't. I'm having issues with the ProtectedBranches.edit function kicking back a 400 Bad Request.

I patched in the branch and inspected the request and it honestly looks fine from what I can tell. But, when I do the exact same request in curl it works just fine :-/

[jdalrymple]

Hmm, thats odd, any chance you could recreate the problem in a public repo so i could debug later?

[trevor-vaughan]

I'll see what I can do. I'm going to try rolling back a few versions and see if it works (it certainly used to).

[jdalrymple]

Def share if you find anything!

[trevor-vaughan]

Looks like the last time it worked was back in @gitbeaker/node 35.8.0

[jdalrymple]

haha oh damn. big changes since then. Very possible its related to native fetch in some way, but i wouldn't know without digging deeper

[trevor-vaughan]

I tried passing everything into the raw requester with the same result.

🤔 Could it be related to patch? The other functions on that endpoint work just fine.

[marcelstoer]

@trevor-vaughan I had cases with GitBeaker(Rest) on Node 18 failing due to fetch instabilities. Using Node 20 instead helped massively.

[trevor-vaughan]

I hacked around it using straight got.patch and that worked but :-(. 🤔 I recently moved off of node-fetch for 16 => 18 compat. I'll try using node-fetch again.

[trevor-vaughan]

🤦 Welp, @marcelstoer You hit the nail on the head. Overriding the inbuilt fetch with node-fetch worked. 💀

[jdalrymple]

Ugh, of course 🙃

[trevor-vaughan]

For anyone else that hits this, here's a horrible hack-around for the entry-point to your app:

const semver = require('semver')
if ( semver.lt(process.version, '20.0.0') ) {
  global.fetch = require('node-fetch')
}

[jdalrymple]

Hopefully those hiccups get sorted as time goes on. Id like to know the particular reason why its getting a bad request though. There must be a difference in the request itself thats causing that

[trevor-vaughan]

FWIW, it was only on patch. Everything else was fine. Weird that it worked via got though :-/

[marcelstoer]

I hate to divert even further from the OT (which is wire logging) but for the Node 18 fetch() discussion this might be indeed relevant: nodejs/undici#583 (comment)

[JounQin]

The current logs seem quite useful, a built-in tracing feature would be great for cases like un-ts/changesets-gitlab#122.

The documentation https://www.npmjs.com/package/@gitbeaker/rest#error-handling is still unclear to me. What the error interface would be?

Is the following I can expect?

export interface GitLabAPIError extends Error {
  cause: {
    text: string;
    response: GitlabAPIResponse;
  }
}

[jdalrymple]

Yea, I'll update the docs to show the proper type, that would be more useful.

The cause is an object with the description as a string, and the response as the original Response object from fetch (https://nodejs.org/dist/latest-v18.x/docs/api/globals.html#response)

[jdalrymple]

@JounQin were you able to sort out the source of the issue?

[JounQin]

@jdalrymple Sorry but I'm not quite sure to understand what is your meaning. I'm just trying get more useful error details.

See also un-ts/changesets-gitlab#122

[jdalrymple]

@jdalrymple Sorry but I'm not quite sure to understand what is your meaning. I'm just trying get more useful error details.

See also un-ts/changesets-gitlab#122

I responded in your pr 😌