Inconsistency in the test result using lighthouse programatically vs in devtool

[amrsa1]

FAQ

• Yes, my issue is not about variability or throttling.
• Yes, my issue is not about a specific accessibility audit (file with axe-core instead).
• Yes, my issue is not answered by other FAQs.

URL

any

What happened?

i was experimenting the lighthouse library and checking the html report against a url, but when i run the analysis from lighthouse in devtool im getting complete different results, which make one of the results is unreliable not sure now which one is supposed to be the accurate one.

result from library

result from lighthouse in devtool

What did you expect?

same results are close results

import fs from "fs";
import lighthouse from "lighthouse";
import * as chromeLauncher from "chrome-launcher";

const chrome = await chromeLauncher.launch({ chromeFlags: ["--headless"] });
const runnerResult = await lighthouse(
  "example",
  {
    logLevel: "error",
    output: "html",
    onlyCategories: ["performance", "accessibility", "best-practices",],
    formFactor: "desktop",
    port: chrome.port,
    screenEmulation: {
      disabled: true,
    },
  }
);

// `.report` is the HTML report as a string
const reportHtml = runnerResult.report;
fs.writeFileSync("lhreport.html", reportHtml);

// `.lhr` is the Lighthouse Result as a JS object
console.log("Report is done for", runnerResult.lhr.finalDisplayedUrl);
console.log(
  "Performance score was",
  runnerResult.lhr.categories.performance.score * 100
);

chrome.kill();

What have you tried?

No response

How were you running Lighthouse?

node

Lighthouse Version

12.2.2

Chrome Version

Chromium: 131.0.6778.108

Node Version

22

OS

No response

Relevant log output

No response

[adamraine]

Please see our variability docs: https://github.com/GoogleChrome/lighthouse/blob/main/docs/variability.md

[amrsa1]

Hmm doesnt make sense as im testing against same server results using library are similar after multiple run to each others

And results from dev tools are similar.

But node library results are not even close to devtool results, we are speaking about 50% variation in result which make the results unreliable which one is the true one ?!