You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello, I am building a crawler with go-rod and I want to have a clear state for every request. The easiest solution I found was to create an incognito browser instance every time
func (c *crawler) Clean(ctx context.Context) error {
if err := c.browser.Close(); err != nil {
return fmt.Errorf("failed to close browser: %w", err)
}
newBrowser, err := c.parentBrowser.Incognito()
if err != nil {
return fmt.Errorf("failed to create incognito browser: %w", err)
}
c.browser = newBrowser
return nil
}
parentBrowser is the actual browser instance browser is the browser we're using for the actual crawling
This cleaning isn't happening too often. But I started experiencing connectivity issues when adding this functionality
failed to get page, cause: write tcp 192.168.155.12:3000->192.168.155.18:7317: use of closed network connection
Could this be related? Or is this some unfortunate timing of infrastructure issues? My suspicion was that the browser might not be initialized or some weird caching is happening, but that doesn't seem to be case as every follow-up request works fine
For every instance of crawler I spin up a new browser via a ManagedLauncher but from the docs for TargetCreateBrowserContext (experimental) it states Creates a new empty BrowserContext. Similar to an incognito profile but you can have more than one., so basically I can use a single browser instance and create new Incognito browsers from it, and they'll all be independent from each-other in terms of cache, cookies and such?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Hello, I am building a crawler with
go-rod
and I want to have a clear state for every request. The easiest solution I found was to create an incognito browser instance every timeparentBrowser
is the actual browser instancebrowser
is the browser we're using for the actual crawlingThis cleaning isn't happening too often. But I started experiencing connectivity issues when adding this functionality
ManagedLauncher
but from the docs forTargetCreateBrowserContext (experimental)
it statesCreates a new empty BrowserContext. Similar to an incognito profile but you can have more than one.
, so basically I can use a single browser instance and create newIncognito
browsers from it, and they'll all be independent from each-other in terms of cache, cookies and such?Beta Was this translation helpful? Give feedback.
All reactions