Here's a more minimal repro:

function Component() {}
function Shape() {}

function Box() {}
Box.prototype = Object.create(Shape.prototype);
Box.prototype.constructor = Box;

function checkInstanceOf(a, b) {
  return a instanceof b;
}

console.log(
  'Box.prototype instanceof Component:',
  Box.prototype instanceof Component);

console.log('checkInstanceOf:',
  checkInstanceOf(Box.prototype, Component));

Object.setPrototypeOf(Shape.prototype, Component.prototype);

console.log(
  'Box.prototype instanceof Component:',
  Box.prototype instanceof Component);

console.log(
  'checkInstanceOf:',
  checkInstanceOf(Box.prototype, Component));

Results

#### ch
Box.prototype instanceof Component: false
checkInstanceOf: false
Box.prototype instanceof Component: true
checkInstanceOf: false

#### v8
Box.prototype instanceof Component: false
checkInstanceOf: false
Box.prototype instanceof Component: true
checkInstanceOf: true

It appears that the second call to checkInstanceOf assumes that the prototype chain has not been modified from the first call.

I've had a quick look setPrototypeOf does not appear to invalidate the IsInst inline cache and so this case involves an incorrect cache hit.

Unfortunately I cannot see an easy way to access the IsInst inline cache from setPrototype of.

I'd like to tag this for v1.12 BUT I can't work out how to fix it - if anyone would like to dig into this that would be great.

What about ...?

scriptContext->GetThreadContext()->InvalidateAllIsInstInlineCaches();

What about ...?

scriptContext->GetThreadContext()->InvalidateAllIsInstInlineCaches();

This will invalidate all caches, not just needed ones, so this is a pretty expensive operation to do

Having had a look, it seems currently IsInst inline caches are stored by function, not by object.

InvalidateAllIsInstInlineCaches would dump all of them for all functions whether they relate to the object we're looking at or not.

The method to dump a single one is ThreadContext::UnregisterIsInstInlineCache(Js::IsInstInlineCache * inlineCache, Js::Var function) to use this correctly we need to get pointers to the caches we need to invalidate AND for each one, a pointer to the function it relates to.

Without creating a new structure the only way to do this would be to iterate over isInstInlineCacheByFunction (as is done in InvalidateAllIsInstInlineCaches) AND then for each one, iterate over its list (as is done in InvalidateIsInstInlineCacheList) but then put in a check to see if it's referencing the Type we're looking at and invalidate it (with UnregisterIsInstInlineCache) if it is NOTE we'd have to use UnregisterIsInstInlineCache rather than a memset because that moves up the next element rather than creating a hole in the list.

This me a bit messy and involve a lot of iteration and checks BUT no one should be using a change prototype method on a fast path so I figure it's ok. Note we'd need to do not just the object who's prototype is being changed BUT any that inherit from it.