Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Argus] heap out of memory crash #5080

Open
kdembler opened this issue Feb 21, 2024 · 1 comment
Open

[Argus] heap out of memory crash #5080

kdembler opened this issue Feb 21, 2024 · 1 comment
Labels
argus Argus distributor node bug Something isn't working

Comments

@kdembler
Copy link
Member

kdembler commented Feb 21, 2024 •
edited
Loading

Full log with crash at the end:

2024-02-21 17:45:51:4551 PublicApi http: HTTP GET /api/v1/assets/2101959
{
    "meta": {
        "req": {
            "url": "/api/v1/assets/2101959",
            "headers": {
                "host": "dist1.joyutils.org",
                "user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
                "accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
                "accept-encoding": "gzip",
                "accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
                "cdn-loop": "cloudflare",
                "cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
                "cf-ipcountry": "AR",
                "cf-ray": "8590cd7e2922ab32-EZE",
                "cf-visitor": "{\"scheme\":\"https\"}",
                "referer": "https://gleev.xyz/",
                "sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
                "sec-ch-ua-mobile": "?1",
                "sec-ch-ua-platform": "\"Android\"",
                "sec-fetch-dest": "image",
                "sec-fetch-mode": "no-cors",
                "sec-fetch-site": "cross-site",
                "traceparent": "00-5ccdb8e7a7da8b8a17c9639cdfa34827-d6262f86b925a15e-01",
                "x-forwarded-for": "172.69.35.139",
                "x-forwarded-host": "dist1.joyutils.org",
                "x-forwarded-proto": "https"
            },
            "method": "GET",
            "httpVersion": "1.1",
            "originalUrl": "/api/v1/assets/2101959",
            "query": {}
        },
        "res": {
            "headers": {
                "x-powered-by": "Express",
                "access-control-allow-origin": "*",
                "timing-allow-origin": "*",
                "access-control-expose-headers": "x-cache, x-data-source",
                "content-type": "application/json; charset=utf-8",
                "content-length": "105",
                "etag": "W/\"69-YoCNAtC7v+WLZhCrBKuH2VsJUis\""
            },
            "statusCode": 500
        },
        "responseTime": 27719,
        "prematurelyClosed": false
    },
    "trace_id": "5ccdb8e7a7da8b8a17c9639cdfa34827",
    "span_id": "b9b56f20f279f540",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://andybut.xyz/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
    "span_id": "5f669cb66119a1e9",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://andybut.xyz/storage/api/v1"
    },
    "objectId": "2101915",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
    "span_id": "5f669cb66119a1e9",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://stor1.joyutils.org/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
    "span_id": "5f669cb66119a1e9",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://stor1.joyutils.org/storage/api/v1"
    },
    "objectId": "2101915",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
    "span_id": "5f669cb66119a1e9",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://joystream.joystream-arsi44.space/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
    "span_id": "5f669cb66119a1e9",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://joystream.joystream-arsi44.space/storage/api/v1"
    },
    "objectId": "2101915",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
    "span_id": "5f669cb66119a1e9",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://storage.mrbovo.xyz/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
    "span_id": "5f669cb66119a1e9",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://storage.mrbovo.xyz/storage/api/v1"
    },
    "objectId": "2101915",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
    "span_id": "5f669cb66119a1e9",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 NetworkingManager error: Data object download failed
{
    "err": {
        "message": "Failed to download object 2101915 from any availablable storage provider",
        "stack": "Error: Failed to download object 2101915 from any availablable storage provider\n    at fail (/joystream/distributor-node/lib/services/networking/NetworkingService.js:219:24)\n    at Queue.<anonymous> (/joystream/distributor-node/lib/services/networking/NetworkingService.js:260:21)\n    at Queue.emit (node:events:517:28)\n    at Queue.done (/joystream/node_modules/queue/index.js:194:8)\n    at next (/joystream/node_modules/queue/index.js:118:16)\n    at /joystream/node_modules/queue/index.js:150:14\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at runNextTicks (node:internal/process/task_queues:64:3)\n    at listOnTimeout (node:internal/timers:538:9)\n    at process.processTimers (node:internal/timers:512:7)"
    },
    "trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
    "span_id": "5f669cb66119a1e9",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 PublicApi error: middlewareError
{
    "err": {
        "message": "Failed to download object 2101915 from any availablable storage provider",
        "stack": "Error: Failed to download object 2101915 from any availablable storage provider\n    at fail (/joystream/distributor-node/lib/services/networking/NetworkingService.js:218:25)\n    at Queue.<anonymous> (/joystream/distributor-node/lib/services/networking/NetworkingService.js:260:21)\n    at Queue.emit (node:events:517:28)\n    at Queue.done (/joystream/node_modules/queue/index.js:194:8)\n    at next (/joystream/node_modules/queue/index.js:118:16)\n    at /joystream/node_modules/queue/index.js:150:14\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at runNextTicks (node:internal/process/task_queues:64:3)\n    at listOnTimeout (node:internal/timers:538:9)\n    at process.processTimers (node:internal/timers:512:7)"
    },
    "req": {
        "url": "/api/v1/assets/2101915",
        "headers": {
            "host": "dist1.joyutils.org",
            "user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
            "accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
            "accept-encoding": "gzip",
            "accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
            "cdn-loop": "cloudflare",
            "cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
            "cf-ipcountry": "AR",
            "cf-ray": "8590cd7e2925ab32-EZE",
            "cf-visitor": "{\"scheme\":\"https\"}",
            "referer": "https://gleev.xyz/",
            "sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
            "sec-ch-ua-mobile": "?1",
            "sec-ch-ua-platform": "\"Android\"",
            "sec-fetch-dest": "image",
            "sec-fetch-mode": "no-cors",
            "sec-fetch-site": "cross-site",
            "traceparent": "00-a93216e4db2d4bb4c93708c9b31c5b71-0bd49bf08deb91aa-01",
            "x-forwarded-for": "172.69.35.138",
            "x-forwarded-host": "dist1.joyutils.org",
            "x-forwarded-proto": "https"
        },
        "method": "GET",
        "httpVersion": "1.1",
        "originalUrl": "/api/v1/assets/2101915",
        "query": {}
    },
    "trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
    "span_id": "5f669cb66119a1e9",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 PublicApi http: HTTP GET /api/v1/assets/2101915
{
    "meta": {
        "req": {
            "url": "/api/v1/assets/2101915",
            "headers": {
                "host": "dist1.joyutils.org",
                "user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
                "accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
                "accept-encoding": "gzip",
                "accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
                "cdn-loop": "cloudflare",
                "cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
                "cf-ipcountry": "AR",
                "cf-ray": "8590cd7e2925ab32-EZE",
                "cf-visitor": "{\"scheme\":\"https\"}",
                "referer": "https://gleev.xyz/",
                "sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
                "sec-ch-ua-mobile": "?1",
                "sec-ch-ua-platform": "\"Android\"",
                "sec-fetch-dest": "image",
                "sec-fetch-mode": "no-cors",
                "sec-fetch-site": "cross-site",
                "traceparent": "00-a93216e4db2d4bb4c93708c9b31c5b71-0bd49bf08deb91aa-01",
                "x-forwarded-for": "172.69.35.138",
                "x-forwarded-host": "dist1.joyutils.org",
                "x-forwarded-proto": "https"
            },
            "method": "GET",
            "httpVersion": "1.1",
            "originalUrl": "/api/v1/assets/2101915",
            "query": {}
        },
        "res": {
            "headers": {
                "x-powered-by": "Express",
                "access-control-allow-origin": "*",
                "timing-allow-origin": "*",
                "access-control-expose-headers": "x-cache, x-data-source",
                "content-type": "application/json; charset=utf-8",
                "content-length": "105",
                "etag": "W/\"69-17VO0ULVLc30wSWRZevqEyq7CAk\""
            },
            "statusCode": 500
        },
        "responseTime": 27720,
        "prematurelyClosed": false
    },
    "trace_id": "a93216e4db2d4bb4c93708c9b31c5b71",
    "span_id": "5f669cb66119a1e9",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://andybut.xyz/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "d1fcecc237c7860294550a8740333cf0",
    "span_id": "0b96bb65ac3d7e19",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://andybut.xyz/storage/api/v1"
    },
    "objectId": "2101979",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "d1fcecc237c7860294550a8740333cf0",
    "span_id": "0b96bb65ac3d7e19",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://stor1.joyutils.org/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "d1fcecc237c7860294550a8740333cf0",
    "span_id": "0b96bb65ac3d7e19",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://stor1.joyutils.org/storage/api/v1"
    },
    "objectId": "2101979",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "d1fcecc237c7860294550a8740333cf0",
    "span_id": "0b96bb65ac3d7e19",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://joystream.joystream-arsi44.space/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "d1fcecc237c7860294550a8740333cf0",
    "span_id": "0b96bb65ac3d7e19",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://joystream.joystream-arsi44.space/storage/api/v1"
    },
    "objectId": "2101979",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "d1fcecc237c7860294550a8740333cf0",
    "span_id": "0b96bb65ac3d7e19",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://storage.mrbovo.xyz/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "d1fcecc237c7860294550a8740333cf0",
    "span_id": "0b96bb65ac3d7e19",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://storage.mrbovo.xyz/storage/api/v1"
    },
    "objectId": "2101979",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "d1fcecc237c7860294550a8740333cf0",
    "span_id": "0b96bb65ac3d7e19",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 NetworkingManager error: Data object download failed
{
    "err": {
        "message": "Failed to download object 2101979 from any availablable storage provider",
        "stack": "Error: Failed to download object 2101979 from any availablable storage provider\n    at fail (/joystream/distributor-node/lib/services/networking/NetworkingService.js:219:24)\n    at Queue.<anonymous> (/joystream/distributor-node/lib/services/networking/NetworkingService.js:260:21)\n    at Queue.emit (node:events:517:28)\n    at Queue.done (/joystream/node_modules/queue/index.js:194:8)\n    at next (/joystream/node_modules/queue/index.js:118:16)\n    at /joystream/node_modules/queue/index.js:150:14\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at runNextTicks (node:internal/process/task_queues:64:3)\n    at listOnTimeout (node:internal/timers:538:9)\n    at process.processTimers (node:internal/timers:512:7)"
    },
    "trace_id": "d1fcecc237c7860294550a8740333cf0",
    "span_id": "0b96bb65ac3d7e19",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 PublicApi error: middlewareError
{
    "err": {
        "message": "Failed to download object 2101979 from any availablable storage provider",
        "stack": "Error: Failed to download object 2101979 from any availablable storage provider\n    at fail (/joystream/distributor-node/lib/services/networking/NetworkingService.js:218:25)\n    at Queue.<anonymous> (/joystream/distributor-node/lib/services/networking/NetworkingService.js:260:21)\n    at Queue.emit (node:events:517:28)\n    at Queue.done (/joystream/node_modules/queue/index.js:194:8)\n    at next (/joystream/node_modules/queue/index.js:118:16)\n    at /joystream/node_modules/queue/index.js:150:14\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at runNextTicks (node:internal/process/task_queues:64:3)\n    at listOnTimeout (node:internal/timers:538:9)\n    at process.processTimers (node:internal/timers:512:7)"
    },
    "req": {
        "url": "/api/v1/assets/2101979",
        "headers": {
            "host": "dist1.joyutils.org",
            "user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
            "accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
            "accept-encoding": "gzip",
            "accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
            "cdn-loop": "cloudflare",
            "cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
            "cf-ipcountry": "AR",
            "cf-ray": "8590cd7e2912ab32-EZE",
            "cf-visitor": "{\"scheme\":\"https\"}",
            "referer": "https://gleev.xyz/",
            "sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
            "sec-ch-ua-mobile": "?1",
            "sec-ch-ua-platform": "\"Android\"",
            "sec-fetch-dest": "image",
            "sec-fetch-mode": "no-cors",
            "sec-fetch-site": "cross-site",
            "traceparent": "00-d1fcecc237c7860294550a8740333cf0-91e764aa10290763-01",
            "x-forwarded-for": "172.69.35.139",
            "x-forwarded-host": "dist1.joyutils.org",
            "x-forwarded-proto": "https"
        },
        "method": "GET",
        "httpVersion": "1.1",
        "originalUrl": "/api/v1/assets/2101979",
        "query": {}
    },
    "trace_id": "d1fcecc237c7860294550a8740333cf0",
    "span_id": "0b96bb65ac3d7e19",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 PublicApi http: HTTP GET /api/v1/assets/2101979
{
    "meta": {
        "req": {
            "url": "/api/v1/assets/2101979",
            "headers": {
                "host": "dist1.joyutils.org",
                "user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
                "accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
                "accept-encoding": "gzip",
                "accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
                "cdn-loop": "cloudflare",
                "cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
                "cf-ipcountry": "AR",
                "cf-ray": "8590cd7e2912ab32-EZE",
                "cf-visitor": "{\"scheme\":\"https\"}",
                "referer": "https://gleev.xyz/",
                "sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
                "sec-ch-ua-mobile": "?1",
                "sec-ch-ua-platform": "\"Android\"",
                "sec-fetch-dest": "image",
                "sec-fetch-mode": "no-cors",
                "sec-fetch-site": "cross-site",
                "traceparent": "00-d1fcecc237c7860294550a8740333cf0-91e764aa10290763-01",
                "x-forwarded-for": "172.69.35.139",
                "x-forwarded-host": "dist1.joyutils.org",
                "x-forwarded-proto": "https"
            },
            "method": "GET",
            "httpVersion": "1.1",
            "originalUrl": "/api/v1/assets/2101979",
            "query": {}
        },
        "res": {
            "headers": {
                "x-powered-by": "Express",
                "access-control-allow-origin": "*",
                "timing-allow-origin": "*",
                "access-control-expose-headers": "x-cache, x-data-source",
                "content-type": "application/json; charset=utf-8",
                "content-length": "105",
                "etag": "W/\"69-GuyLFrfFSeAf+cgtCn08QrrAopo\""
            },
            "statusCode": 500
        },
        "responseTime": 27726,
        "prematurelyClosed": false
    },
    "trace_id": "d1fcecc237c7860294550a8740333cf0",
    "span_id": "0b96bb65ac3d7e19",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://andybut.xyz/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "967d1636c501a5f625c64943e74227dc",
    "span_id": "1c64976e266f47d3",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://andybut.xyz/storage/api/v1"
    },
    "objectId": "2101965",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "967d1636c501a5f625c64943e74227dc",
    "span_id": "1c64976e266f47d3",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://stor1.joyutils.org/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "967d1636c501a5f625c64943e74227dc",
    "span_id": "1c64976e266f47d3",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://stor1.joyutils.org/storage/api/v1"
    },
    "objectId": "2101965",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "967d1636c501a5f625c64943e74227dc",
    "span_id": "1c64976e266f47d3",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://joystream.joystream-arsi44.space/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "967d1636c501a5f625c64943e74227dc",
    "span_id": "1c64976e266f47d3",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://joystream.joystream-arsi44.space/storage/api/v1"
    },
    "objectId": "2101965",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "967d1636c501a5f625c64943e74227dc",
    "span_id": "1c64976e266f47d3",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://storage.mrbovo.xyz/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "967d1636c501a5f625c64943e74227dc",
    "span_id": "1c64976e266f47d3",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://storage.mrbovo.xyz/storage/api/v1"
    },
    "objectId": "2101965",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "967d1636c501a5f625c64943e74227dc",
    "span_id": "1c64976e266f47d3",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 NetworkingManager error: Data object download failed
{
    "err": {
        "message": "Failed to download object 2101965 from any availablable storage provider",
        "stack": "Error: Failed to download object 2101965 from any availablable storage provider\n    at fail (/joystream/distributor-node/lib/services/networking/NetworkingService.js:219:24)\n    at Queue.<anonymous> (/joystream/distributor-node/lib/services/networking/NetworkingService.js:260:21)\n    at Queue.emit (node:events:517:28)\n    at Queue.done (/joystream/node_modules/queue/index.js:194:8)\n    at next (/joystream/node_modules/queue/index.js:118:16)\n    at /joystream/node_modules/queue/index.js:150:14\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at runNextTicks (node:internal/process/task_queues:64:3)\n    at listOnTimeout (node:internal/timers:538:9)\n    at process.processTimers (node:internal/timers:512:7)"
    },
    "trace_id": "967d1636c501a5f625c64943e74227dc",
    "span_id": "1c64976e266f47d3",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 PublicApi error: middlewareError
{
    "err": {
        "message": "Failed to download object 2101965 from any availablable storage provider",
        "stack": "Error: Failed to download object 2101965 from any availablable storage provider\n    at fail (/joystream/distributor-node/lib/services/networking/NetworkingService.js:218:25)\n    at Queue.<anonymous> (/joystream/distributor-node/lib/services/networking/NetworkingService.js:260:21)\n    at Queue.emit (node:events:517:28)\n    at Queue.done (/joystream/node_modules/queue/index.js:194:8)\n    at next (/joystream/node_modules/queue/index.js:118:16)\n    at /joystream/node_modules/queue/index.js:150:14\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at runNextTicks (node:internal/process/task_queues:64:3)\n    at listOnTimeout (node:internal/timers:538:9)\n    at process.processTimers (node:internal/timers:512:7)"
    },
    "req": {
        "url": "/api/v1/assets/2101965",
        "headers": {
            "host": "dist1.joyutils.org",
            "user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
            "accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
            "accept-encoding": "gzip",
            "accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
            "cdn-loop": "cloudflare",
            "cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
            "cf-ipcountry": "AR",
            "cf-ray": "8590cd7e292dab32-EZE",
            "cf-visitor": "{\"scheme\":\"https\"}",
            "referer": "https://gleev.xyz/",
            "sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
            "sec-ch-ua-mobile": "?1",
            "sec-ch-ua-platform": "\"Android\"",
            "sec-fetch-dest": "image",
            "sec-fetch-mode": "no-cors",
            "sec-fetch-site": "cross-site",
            "traceparent": "00-967d1636c501a5f625c64943e74227dc-74de9b66e6252cb1-01",
            "x-forwarded-for": "172.69.35.139",
            "x-forwarded-host": "dist1.joyutils.org",
            "x-forwarded-proto": "https"
        },
        "method": "GET",
        "httpVersion": "1.1",
        "originalUrl": "/api/v1/assets/2101965",
        "query": {}
    },
    "trace_id": "967d1636c501a5f625c64943e74227dc",
    "span_id": "1c64976e266f47d3",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 PublicApi http: HTTP GET /api/v1/assets/2101965
{
    "meta": {
        "req": {
            "url": "/api/v1/assets/2101965",
            "headers": {
                "host": "dist1.joyutils.org",
                "user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
                "accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
                "accept-encoding": "gzip",
                "accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
                "cdn-loop": "cloudflare",
                "cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
                "cf-ipcountry": "AR",
                "cf-ray": "8590cd7e292dab32-EZE",
                "cf-visitor": "{\"scheme\":\"https\"}",
                "referer": "https://gleev.xyz/",
                "sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
                "sec-ch-ua-mobile": "?1",
                "sec-ch-ua-platform": "\"Android\"",
                "sec-fetch-dest": "image",
                "sec-fetch-mode": "no-cors",
                "sec-fetch-site": "cross-site",
                "traceparent": "00-967d1636c501a5f625c64943e74227dc-74de9b66e6252cb1-01",
                "x-forwarded-for": "172.69.35.139",
                "x-forwarded-host": "dist1.joyutils.org",
                "x-forwarded-proto": "https"
            },
            "method": "GET",
            "httpVersion": "1.1",
            "originalUrl": "/api/v1/assets/2101965",
            "query": {}
        },
        "res": {
            "headers": {
                "x-powered-by": "Express",
                "access-control-allow-origin": "*",
                "timing-allow-origin": "*",
                "access-control-expose-headers": "x-cache, x-data-source",
                "content-type": "application/json; charset=utf-8",
                "content-length": "105",
                "etag": "W/\"69-e0TQIzyfoufzzJX6e7Fyy/VdXdA\""
            },
            "statusCode": 500
        },
        "responseTime": 27739,
        "prematurelyClosed": false
    },
    "trace_id": "967d1636c501a5f625c64943e74227dc",
    "span_id": "1c64976e266f47d3",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://andybut.xyz/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "06efab4bfed59b61dade00ff424919e7",
    "span_id": "76c73c02df4913f4",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://andybut.xyz/storage/api/v1"
    },
    "objectId": "2101963",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "06efab4bfed59b61dade00ff424919e7",
    "span_id": "76c73c02df4913f4",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://stor1.joyutils.org/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "06efab4bfed59b61dade00ff424919e7",
    "span_id": "76c73c02df4913f4",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://stor1.joyutils.org/storage/api/v1"
    },
    "objectId": "2101963",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "06efab4bfed59b61dade00ff424919e7",
    "span_id": "76c73c02df4913f4",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://joystream.joystream-arsi44.space/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "06efab4bfed59b61dade00ff424919e7",
    "span_id": "76c73c02df4913f4",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://joystream.joystream-arsi44.space/storage/api/v1"
    },
    "objectId": "2101963",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "06efab4bfed59b61dade00ff424919e7",
    "span_id": "76c73c02df4913f4",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Request timeout of 5000ms reached
{
    "0": {
        "endpoint": "https://storage.mrbovo.xyz/storage/api/v1"
    },
    "timeoutMs": 5000,
    "trace_id": "06efab4bfed59b61dade00ff424919e7",
    "span_id": "76c73c02df4913f4",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 StorageNodeApi error: Unexpected error while requesting data object
{
    "0": {
        "endpoint": "https://storage.mrbovo.xyz/storage/api/v1"
    },
    "objectId": "2101963",
    "err": {
        "message": "Request timeout"
    },
    "trace_id": "06efab4bfed59b61dade00ff424919e7",
    "span_id": "76c73c02df4913f4",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 NetworkingManager error: Data object download failed
{
    "err": {
        "message": "Failed to download object 2101963 from any availablable storage provider",
        "stack": "Error: Failed to download object 2101963 from any availablable storage provider\n    at fail (/joystream/distributor-node/lib/services/networking/NetworkingService.js:219:24)\n    at Queue.<anonymous> (/joystream/distributor-node/lib/services/networking/NetworkingService.js:260:21)\n    at Queue.emit (node:events:517:28)\n    at Queue.done (/joystream/node_modules/queue/index.js:194:8)\n    at next (/joystream/node_modules/queue/index.js:118:16)\n    at /joystream/node_modules/queue/index.js:150:14\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at runNextTicks (node:internal/process/task_queues:64:3)\n    at listOnTimeout (node:internal/timers:538:9)\n    at process.processTimers (node:internal/timers:512:7)"
    },
    "trace_id": "06efab4bfed59b61dade00ff424919e7",
    "span_id": "76c73c02df4913f4",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 PublicApi error: middlewareError
{
    "err": {
        "message": "Failed to download object 2101963 from any availablable storage provider",
        "stack": "Error: Failed to download object 2101963 from any availablable storage provider\n    at fail (/joystream/distributor-node/lib/services/networking/NetworkingService.js:218:25)\n    at Queue.<anonymous> (/joystream/distributor-node/lib/services/networking/NetworkingService.js:260:21)\n    at Queue.emit (node:events:517:28)\n    at Queue.done (/joystream/node_modules/queue/index.js:194:8)\n    at next (/joystream/node_modules/queue/index.js:118:16)\n    at /joystream/node_modules/queue/index.js:150:14\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at runNextTicks (node:internal/process/task_queues:64:3)\n    at listOnTimeout (node:internal/timers:538:9)\n    at process.processTimers (node:internal/timers:512:7)"
    },
    "req": {
        "url": "/api/v1/assets/2101963",
        "headers": {
            "host": "dist1.joyutils.org",
            "user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
            "accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
            "accept-encoding": "gzip",
            "accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
            "cdn-loop": "cloudflare",
            "cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
            "cf-ipcountry": "AR",
            "cf-ray": "8590cd7c5e85ab32-EZE",
            "cf-visitor": "{\"scheme\":\"https\"}",
            "referer": "https://gleev.xyz/",
            "sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
            "sec-ch-ua-mobile": "?1",
            "sec-ch-ua-platform": "\"Android\"",
            "sec-fetch-dest": "image",
            "sec-fetch-mode": "no-cors",
            "sec-fetch-site": "cross-site",
            "traceparent": "00-06efab4bfed59b61dade00ff424919e7-b880249753ee0cfd-01",
            "x-forwarded-for": "172.69.35.138",
            "x-forwarded-host": "dist1.joyutils.org",
            "x-forwarded-proto": "https"
        },
        "method": "GET",
        "httpVersion": "1.1",
        "originalUrl": "/api/v1/assets/2101963",
        "query": {}
    },
    "trace_id": "06efab4bfed59b61dade00ff424919e7",
    "span_id": "76c73c02df4913f4",
    "trace_flags": "01"
}
2024-02-21 17:45:51:4551 PublicApi http: HTTP GET /api/v1/assets/2101963
{
    "meta": {
        "req": {
            "url": "/api/v1/assets/2101963",
            "headers": {
                "host": "dist1.joyutils.org",
                "user-agent": "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36",
                "accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
                "accept-encoding": "gzip",
                "accept-language": "es-US,es-419;q=0.9,es;q=0.8,en;q=0.7,be;q=0.6",
                "cdn-loop": "cloudflare",
                "cf-connecting-ip": "2800:810:524:12bc:916a:a1b:6ac2:2fcb",
                "cf-ipcountry": "AR",
                "cf-ray": "8590cd7c5e85ab32-EZE",
                "cf-visitor": "{\"scheme\":\"https\"}",
                "referer": "https://gleev.xyz/",
                "sec-ch-ua": "\"Not A(Brand\";v=\"99\", \"Google Chrome\";v=\"121\", \"Chromium\";v=\"121\"",
                "sec-ch-ua-mobile": "?1",
                "sec-ch-ua-platform": "\"Android\"",
                "sec-fetch-dest": "image",
                "sec-fetch-mode": "no-cors",
                "sec-fetch-site": "cross-site",
                "traceparent": "00-06efab4bfed59b61dade00ff424919e7-b880249753ee0cfd-01",
                "x-forwarded-for": "172.69.35.138",
                "x-forwarded-host": "dist1.joyutils.org",
                "x-forwarded-proto": "https"
            },
            "method": "GET",
            "httpVersion": "1.1",
            "originalUrl": "/api/v1/assets/2101963",
            "query": {}
        },
        "res": {
            "headers": {
                "x-powered-by": "Express",
                "access-control-allow-origin": "*",
                "timing-allow-origin": "*",
                "access-control-expose-headers": "x-cache, x-data-source",
                "content-type": "application/json; charset=utf-8",
                "content-length": "105",
                "etag": "W/\"69-tiUt7lt6aYy57dMAQTIHVELHQO4\""
            },
            "statusCode": 500
        },
        "responseTime": 27752,
        "prematurelyClosed": false
    },
    "trace_id": "06efab4bfed59b61dade00ff424919e7",
    "span_id": "76c73c02df4913f4",
    "trace_flags": "01"
}

<--- Last few GCs --->

[7:0x55f3870] 106762850 ms: Scavenge 4039.2 (4114.6) -> 4038.9 (4125.6) MB, 8.6 / 0.0 ms  (average mu = 0.253, current mu = 0.152) allocation failure;
[7:0x55f3870] 106762872 ms: Scavenge 4045.8 (4125.6) -> 4045.6 (4126.4) MB, 14.1 / 0.0 ms  (average mu = 0.253, current mu = 0.152) allocation failure;
[7:0x55f3870] 106764336 ms: Scavenge 4046.6 (4126.4) -> 4045.8 (4148.4) MB, 1463.1 / 0.0 ms  (average mu = 0.253, current mu = 0.152) allocation failure;


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb95b60 node::Abort() [node]
 2: 0xa9a7f8  [node]
 3: 0xd6f2f0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xd6f697 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xf4cba5  [node]
 6: 0xf5f08d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 7: 0xf3978e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 8: 0xf3ab57 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 9: 0xf1bd2a v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
10: 0x12e114f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
11: 0x170deb9  [node]
/joystream/distributor-node/runner.sh: line 8:     7 Aborted                 (core dumped) node --require @joystream/opentelemetry ./bin/run $*
Loaded Application Instrumentation: "Distributor Node"
Starting tracing...
2024-02-21 17:46:38:4638 App info: Starting the app

All of the above logs claim to be at the exact same timestamp which is a bit weird.

CleanShot 2024-02-21 at 19 54 13@2x

Looks like a memory leak, not sure what happened on 19th
@kdembler kdembler added bug Something isn't working argus Argus distributor node labels Feb 21, 2024
@kdembler
Copy link
Member Author

Seems to still be happening on 2.0
CleanShot 2024-02-23 at 11 05 00@2x

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
argus Argus distributor node bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kdembler