Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Network Inspection #75

Open
eugeneo opened this issue Dec 1, 2016 · 65 comments
Open

Support Network Inspection #75

eugeneo opened this issue Dec 1, 2016 · 65 comments

Comments

@eugeneo
Copy link

eugeneo commented Dec 1, 2016

Network domain is what Chrome DevTools (and any other interested frontends) use for inspecting the HTTP and WS traffic in the browser. This document provides some insight into Chrome DevTools UI and here is a description of the protocol for obtaining that data.

We propose to introduce a Node specific agent that would expose the same (or subset) of this protocol to enable the UI. It does not seem possible to share the implementation between Blink and Node.js.

This agent would also introduce an API for the modules to report the events. E.g. WebSocket modules should be able to notify the agent when WS frames are sent or received.

@sidorares
Copy link

would be great to have support for node tcp/udp as well, not sure if that can fit into existing UI and debugger protocol

@jkrems
Copy link
Contributor

jkrems commented Dec 1, 2016

Would it make sense to use trace events as the interface to the agent? That would also provide an interface for 3rd party modules (e.g. Websocket) out-of-the-box.

I'm not sure if tcp/udp would become confusing because it's another level than the others. Would http requests show up twice effectively? But having a story for things like memcached's TCP protocol would be great. The obvious/hacky solution would be for them to pretend to be websocket frames.

@eugeneo
Copy link
Author

eugeneo commented Dec 1, 2016

Looks like trace events might be usable for this. HTTP requests/responses might get quite big - I wonder if there's any limit on event sizes...

+@matthewloring, @ofrobots - would it be possible to use the trace events framework for this? E.g. inspector would have an agent that "listens" on event categories and then sends inspector protocol notifications.

@jkrems
Copy link
Contributor

jkrems commented Dec 2, 2016

One thing I ran into with a similar feature in bugger: Retaining all request and response bodies in the process can become problematic, especially when there's no easy way to disable it. E.g. "hey let me profile this app while sending a couple of thousand of requests through it to find that memory leak" - aaand you're running out of memory because all request/response bodies are being kept in memory by the agent.

@matthewloring
Copy link

There is not currently a mechanism for observing trace events as they are generated. We could explore this in the future once the design in nodejs/node#9304 solidifies.

@alnorris
Copy link

Would it be possible to monkey patch this? Similar to how node inspector does it?

@paulirish
Copy link
Contributor

@alnorris honestly it looks like the existing monkeypatch from node-inspector (Injections/NetworkAgent.js) would get pretty far.

The emitted events would just need to make their way to the InspectorAgent

@joshgav
Copy link
Contributor

joshgav commented Feb 27, 2017

My architecture proposal in nodejs/node#7393 is largely about allowing implementation of Domain.Method dispatchers other than those tied directly to V8, which is all we have now. The Network domain implementation we're discussing in this thread could be handled by just such an alternate dispatcher if our architecture allowed it.

Allowing alternate dispatchers would a) allow complete alternate implementations, e.g. for other runtimes like Chakra or SpiderMonkey; and b) facilitate one-off (monkey-patched?) alternate or additional implementations for specific Domains and Methods.

/cc @kfarnung who's been looking into such a rearchitecture and Inspector's impact on e.g. Node-ChakraCore more deeply.

@joshgav
Copy link
Contributor

joshgav commented May 11, 2017

Perhaps a way to facilitate inspector "addons" would be to update NodeInspectorClient::dispatchMessageFromFrontend to route based on the Domain name. That is, Debugger, Profiler, Runtime, and all the domains handled by V8 would be routed to V8Inspector as they are now; and others like Network could be routed elsewhere.

@joshgav joshgav changed the title Support for HTTP/WS inspection Support Network Inspection May 11, 2017
@joshgav
Copy link
Contributor

joshgav commented May 11, 2017

@eugeneo I changed the title so it would be more obvious what this issue is about, please change back if you disagree. Thanks!

@eugeneo
Copy link
Author

eugeneo commented May 11, 2017

@joshgav I agree. I want to look into JS handlers for the inspector messages. Currently I am waiting for the inspector JS API PR to land.

@jkrems
Copy link
Contributor

jkrems commented Oct 24, 2017

Was there any movement on this? I think it would be a really great feature. It looks like in Chrome there's a pretty straight-forward check (InspectorSession::DispatchProtocolMessage):

void InspectorSession::DispatchProtocolMessage(const String& method,
                                               const String& message) {
  DCHECK(!disposed_);
  if (v8_inspector::V8InspectorSession::canDispatchMethod(
          ToV8InspectorStringView(method))) {
    v8_session_->dispatchProtocolMessage(ToV8InspectorStringView(message));
  } else {
    inspector_backend_dispatcher_->dispatch(
        protocol::StringUtil::parseJSON(message));
  }
}

We could "just" introduce an extension to the existing inspector JS API that allows to register additional handlers. Or, alternatively, a C++ API for the same (might be less confusing when it comes to execution / threads).

The idea here is that we could start with opening up an experimental low-level API and then later decide if and what kind of node-specific dispatchers we'd want.

@eugeneo
Copy link
Author

eugeneo commented Oct 24, 2017

Right now I am working on custom protocol handlers on the Node side. I am implementing "Target" domain (that should allow debugging child processes). "Network" domain can be implemented afterwards, but it would require effort from module writers - e.g. WebSocket implementors will need to specifically report WS frames.

@eugeneo
Copy link
Author

eugeneo commented Oct 24, 2017

@jkrems - this is the Node counterpart - https://github.com/eugeneo/node/blob/df8e41d3be705410e3a3389b8660b3289e25399f/src/inspector_agent.cc#L255

Minor difference is that I am passing unrecognized messages to V8 dispatcher to reuse "method not found" notification.

@kaycebasques
Copy link

Chrome DevTools tech writer here. Just talked to somebody who's using node --inspect to debug his Node server via DevTools. He was looking for the Network panel, and was confused why it was missing. His goal was to debug the network requests that his server was making. Seems like a reasonable use case to me!

@nmiddendorff
Copy link

Any update?

@Kielan
Copy link

Kielan commented Sep 12, 2018

Many of us would love to know, there are two medium articles which did not age particularly well and beyond that not much of a page on the internet to explain YES THIS is how you inspect http requests through node with the same powers as chrome network inspector tab.

this nodejs page in the docs for example references cancelled or unsupported projects for the last 2-3 versions of node https://nodejs.org/en/docs/guides/debugging-getting-started/

How can we users help ensure there is a place on the internet for users to find timely documentation on inspection of http requests and options.

@mcollina
Copy link
Member

@Kielan thanks for weighting in! I don't think it's currently possible to see the HTTP requests in the inspector, so there is no place where this is documented. Possibly, we should document that no, it is not possible to inspect http requests and responses. However, work is being carried on to enable this.

Considering that we can now route trace_events into the inspector (nodejs/node#20608).
@eugeneo would that be enough to add this support to the inspector? Do you need any specific data/in a specific format to be able to render HTTP requests?

@eugeneo
Copy link
Author

eugeneo commented Sep 12, 2018

Chrome has a well-define protocol for inspecting network - https://chromedevtools.github.io/devtools-protocol/tot/Network

It should not be too hard to write a handler that will handle the messages, the bigger problem is gathering the data. This may need an effort from some community packages, e.g. WebSocket packages will need to report events to Inspector, maybe something will need to be done to Express (though I expect HTTP support only needs instrumentation in the standard library)

@mcollina
Copy link
Member

@eugeneo that seem very much focus on outgoing requests and not incoming, i.e. http clients vs http servers. How those will be displayed? Gathering the data would not be very hard with async_hooks or by plugging it into our internals, I can help with that.

@eugeneo
Copy link
Author

eugeneo commented Sep 12, 2018

Chrome protocol records both requests and responses.

@AndreasGassmann
Copy link

I hope this does not get closed because it's still an issue.

Because this is currently not implemented, the easiest way for us to test if we mocked all of the requests in our unit tests is to take the computer that runs the tests offline and see which tests fail because of network errors.

Needless to say, I would rather have a network inspection tab in the devtools...

@jkrems
Copy link
Contributor

jkrems commented Oct 25, 2019

One thing that has been happening is that @khanghoang has started implementing a node Network tab for ndb specifically: GoogleChromeLabs/ndb#282

@OliverJAsh
Copy link

For anyone looking to do this today, you can use Charles proxy. Here's a guide: http://marianna.im/tech/capture-nodejs-traffic-with-charles/.

@jkrems
Copy link
Contributor

jkrems commented Oct 31, 2019

Charles is definitely a great way to inspect network traffic in general and can be a super helpful workaround here!

But it only gets users half-way there. One of the powerful features of the Network tab is that it can tie the requests directly to the exact stack of what caused the request. And it would hopefully also add network info to the timeline, making it possible to combine it with custom marks. Still crossing my fingers that at least the ndb thing works out.

@khanghoang
Copy link

@jkrems Totally agree with you about the benefits of having the network tab inside the debugger. At PayPal, We use GraphQL to talk to a ton of services and we found the network tab is really helpful to debug the requests and responses.

I'm glad that you like my PR for ndb but unfortunately, I don't think it has the chance to get merged. I have been trying to ping the maintainer multiple times but no response.

@naugtur
Copy link
Contributor

naugtur commented Nov 1, 2019

For those who are after the stack traces to the code making outgoing requests, check out the network tool from https://github.com/naugtur/debugging-aid
All you need to do to get it to work is node --require=debugging-aid/network app.js

@github-actions
Copy link

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.

@mturley
Copy link

mturley commented Nov 4, 2020

It would be nice to reopen this, as far as I can tell it's still an issue, and quite a frustrating one at that.

@jkrems
Copy link
Contributor

jkrems commented Nov 4, 2020

It would be nice to reopen this, as far as I can tell it's still an issue, and quite a frustrating one at that.

I think the honest answer is that - bare anybody stepping up - this issue is a "won't fix". This could change in the future but especially in the context of nodejs/diagnostics (a working group), closing this issue expresses what action the working group is taking for the foreseeable future.

It may be worth opening an issue in Chrome devtools or nodejs/node but it's a good chunk of work, so I wouldn't be surprised if the issue just lingers (or is closed) there, too.

@mturley
Copy link

mturley commented Nov 4, 2020

Fair enough, thanks for the reply @jkrems

@saivishnutammineni
Copy link

I had the same requirement to debug the network requests. This becomes a bigger problem when debugging a request requires me to check previous requests made too. So I created a NPM package to show network requests made from NodeJS app in a Chrome Dev tools like UI. Thought it might help someone.

Network Activity Viewer

@gkatsanos
Copy link

Please do this.

@quantuminformation

This comment was marked as abuse.

@github-actions github-actions bot removed the stale label Sep 12, 2023
@kettanaito
Copy link

Sharing my thoughts on this.

How to represent incoming/outgoing traffic?

As @mcollina asked, the Network panel in the browser's DevTools always represents client-side traffic. In Node.js, a server may both received and send requests, which has to be represented in the Network tab somehow.

Proposal: follow @nicoburns suggestion #75 (comment) and add both incoming/outgoing traffic in the Network. I'm not aware if CDP supports custom tags for network entries so we could have a checkbox to filter out different kinds of traffic.

How to record the traffic?

I appreciate different third-party suggestions (proxies/interception libraries) but I believe the most straightforward way to do that would be using node:async_hooks and its HTTPINCOMINGMESSAGE and HTTPOUTGOINGMESSAGE events.

One must consider, however, that if a Node.js process has its request modules patches (e.g. when running tests that use API mocking solutions, those events will likely not be called by async_hooks. I believe there is also a tremendous value in being able to inspect mocked network, but for that to work we'd have to spy on higher-level request modules, like http. Once can use Proxy to do that in the least intrusive way possible.

What's to be done?

I have very little knowledge in the CDP but it looks like this is a rough roadmap:

  1. Node.js has to implement the Network domain of the CDP.
  2. Node.js has to use the Network domain to register outgoing/incoming requests based on the async_hooks listeners (such as invoking .requestWillBeSent() for an requests and .responseReceived() for responses).
  3. Node.js has to differentiate between the outgoing and incoming requests somehow.
  4. Tests.

Questions

  1. @mcollina, to your best knowledge, does async_hooks provide sufficient data to be fed to the Network domain of the devtools? Is there any data that's missing/won't be observable?
  2. I suppose with the WebSocket API landing natively in Node.js it will support async_hooks as well? Otherwise it won't be observable.
  3. How to deal with third-party WebSocket implementations, if at all? Userland will take time to migrate to the standard WebSocket API, and until then, they will rely on Socket, I assume. It's observable in async_hooks, as far as I recall. Does that sound like a reliable strategy for implementing WebSocket support in the Network inspector?

@mcollina
Copy link
Member

  1. yes
  2. if not it can be added
  3. 3rd party would not be supported (or possibly)

tbh I would instrument everything with diagnostic_channel and scoop data from there

Copy link

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.

@github-actions github-actions bot added the stale label Apr 16, 2024
@starball5
Copy link

I would not like this issue to be closed. To that end, I bump.

@github-actions github-actions bot removed the stale label Apr 17, 2024
@GrinZero
Copy link

GrinZero commented Jun 5, 2024

Expect to be able to listen to requests with chrome devtools made by your own program? You can refer to my recently developed open source library: https://github.com/GrinZero/node-network-devtools.
It edit the 'http' and 'https' module from your node program and use CDP to show the info in chrome devtools.


I also hope that the official can support it. It's exhausting to do this myself.

@cola119
Copy link
Member

cola119 commented Jun 26, 2024

I hope nodejs/node#53593 can be a starting point to discuss how we can realize network inspection. If you have any suggestions, please feel free to share them!

@mcollina
Copy link
Member

mcollina commented Jul 1, 2024

Thank you @cola119!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests