Rules stop working on large files

[Helyk]

Hi. I use grepc mostly for reading logs, including live logs. Typical case is that I set color of lines or their parts based on log level of entry (info/warning/error/debug), resulting in most of lines having highlights.

I often notice that rules are simply not applied to a log file that I am reading. In the grepc panel I see them as enabled but having zero matches. Sometimes it helps to disable and re-enable a rule, but when this doesn't help I have to reload whole window (closing and opening a file also does not help). After reloading the window I see that rules are applied and have matches, but switching to another editor tab and back results in zero matches again. This issue does not reproduce reliably but I more often see it occurring than not.

Typical size of logs that I work with is 10k-150k lines. I set "max occurrences" in rules to 1M.

If needed, I can provide data for diagnostics and maybe generate an example log.

[stneveadomi]

Hey, thanks for making an issue!

I believe there are three issues highlighted here that I have noticed:

1. Occurrences being 0 randomly
2. Rules not consistently applying across large allocations
3. Whole extension silently crashing after a large regex operation

I personally have noticed these. Allow me to follow up later, and I definitely want to address these. Let me know if I missed something.

[Helyk]

Thank you for quick response. I don't think you missed anything but there's also another issue #116 that I observe on the same files so I'll link it here just in case if there's any connection.

[stneveadomi]

What might also make my testing easier is if you don't mind sharing some of the rules that exhibit this behavior, and then perhaps some example text that I can make a quick script to generate and tail the file.

To get the rules, simply hit the export button in the top right and you can post the export in a code block.

Or identify a single rule in the view code drop down and send that over.

[stneveadomi]

Multi-line is unsurprisingly a pain in the ass to handle, so I was more so waiting for someone to complain about it to actually get to fixing it. And it appears now is that time for me to get up and finally deal with it haha.

[Helyk]

Okay here's a rule that I believe covers all the problems that I mentioned:

    {
        "id": "c1198cbf-7577-40fe-8601-d846329b6ac4",
        "title": "REPLY FALSE",
        "enabled": true,
        "expanded": false,
        "decorationExpanded": false,
        "beforeExpanded": false,
        "afterExpanded": false,
        "occurrencesExpanded": false,
        "overviewRulerColor": "",
        "overviewRulerLane": 7,
        "maxOccurrences": 1000000,
        "regularExpression": "(\\n.*){3}\"should_reply\":false.*",
        "regularExpressionFlags": "g",
        "includedFiles": "openai.log$",
        "excludedFiles": "",
        "backgroundColor": "",
        "outline": "",
        "outlineColor": "",
        "outlineWidth": "",
        "border": "",
        "borderColor": "",
        "borderWidth": "",
        "fontStyle": "",
        "fontWeight": "400",
        "textDecoration": "",
        "cursor": "",
        "color": "#5a6185",
        "isWholeLine": false,
        "before": {},
        "after": {
            "contentText": "",
            "border": "",
            "borderColor": "",
            "fontStyle": "",
            "fontWeight": "",
            "textDecoration": "",
            "color": "#ffebeb",
            "backgroundColor": "",
            "margin": "",
            "width": "",
            "height": ""
        }
    }

The line that triggers this rule together with two preceding lines look like this:

2024/10/24 15:27:01 api.go:462: Consumed OpenAI tokens: 288 + 37 = 325 ($0.000065) on model 'gpt-4o-mini-2024-07-18' in 817.09988ms
2024/10/24 15:27:01 api.go:543: OpenAI called functions:
{Name:should_reply Arguments:{"should_reply":false,"reasoning":"The last message was a general greeting and not specifically directed. Therefore, it's not necessary to reply."}}

Such lines comprise more than a half of the log in question. As you can guess, the other lines that do not trigger this rule generally look like this:

YYYY/MM/DD hh:mm:ss file:line: Some text here

[Helyk]

Just in case, here's also a set of rules that I use for highlighting log levels:

[
    {
        "id": "8dfb588f-f370-4b15-b3e1-b801402c29b9",
        "title": "LOG-ERR",
        "enabled": true,
        "expanded": false,
        "decorationExpanded": false,
        "occurrencesExpanded": false,
        "overviewRulerColor": "#FF5370",
        "overviewRulerLane": "4",
        "maxOccurrences": 1000000,
        "regularExpression": "\"(|\\d_)level\":\"error\"",
        "regularExpressionFlags": "g",
        "includedFiles": "\\.log",
        "excludedFiles": "",
        "backgroundColor": "",
        "outline": "",
        "outlineColor": "",
        "outlineWidth": "",
        "border": "",
        "borderColor": "",
        "borderWidth": "",
        "fontStyle": "",
        "fontWeight": "400",
        "textDecoration": "",
        "cursor": "",
        "color": "#ff5370",
        "isWholeLine": false
    },
    {
        "id": "ec30a261-1f52-40c6-af72-56681bf311aa",
        "title": "LOG-WARN",
        "enabled": true,
        "expanded": false,
        "decorationExpanded": false,
        "occurrencesExpanded": false,
        "overviewRulerColor": "",
        "overviewRulerLane": 7,
        "maxOccurrences": 1000000,
        "regularExpression": "\"(|\\d_)level\":\"warning\"",
        "regularExpressionFlags": "g",
        "includedFiles": "\\.log",
        "excludedFiles": "",
        "backgroundColor": "",
        "outline": "",
        "outlineColor": "",
        "outlineWidth": "",
        "border": "",
        "borderColor": "",
        "borderWidth": "",
        "fontStyle": "",
        "fontWeight": "400",
        "textDecoration": "",
        "cursor": "",
        "color": "#FFCB6B",
        "isWholeLine": false
    },
    {
        "id": "4bd75064-f48b-4a16-93be-759a64a00a6a",
        "title": "LOG-DEBG",
        "enabled": true,
        "expanded": false,
        "decorationExpanded": false,
        "occurrencesExpanded": false,
        "overviewRulerColor": "",
        "overviewRulerLane": 7,
        "maxOccurrences": 1000000,
        "regularExpression": "\"(|\\d_)level\":\"debug\"",
        "regularExpressionFlags": "g",
        "includedFiles": "\\.log",
        "excludedFiles": "",
        "backgroundColor": "",
        "outline": "",
        "outlineColor": "",
        "outlineWidth": "",
        "border": "",
        "borderColor": "",
        "borderWidth": "",
        "fontStyle": "",
        "fontWeight": "400",
        "textDecoration": "",
        "cursor": "",
        "color": "#818cac",
        "isWholeLine": false
    },
    {
        "id": "983a33cd-2c09-40ac-a198-73bb03dada63",
        "title": "LOG-APP",
        "enabled": true,
        "expanded": false,
        "decorationExpanded": false,
        "occurrencesExpanded": false,
        "overviewRulerColor": "#818cac",
        "overviewRulerLane": "2",
        "maxOccurrences": 1000000,
        "regularExpression": "(Application \\w+ \\S+ \\w+)",
        "regularExpressionFlags": "g",
        "includedFiles": "\\.log",
        "excludedFiles": "",
        "backgroundColor": "#818cac",
        "outline": "",
        "outlineColor": "",
        "outlineWidth": "",
        "border": "",
        "borderColor": "",
        "borderWidth": "",
        "fontStyle": "",
        "fontWeight": "400",
        "textDecoration": "",
        "cursor": "",
        "color": "",
        "isWholeLine": false
    }
]

Sample of log with matches for all cases:

{"0_ts":"2024-10-24 09:30:12.217 +05:00","1_level":"info",   "3_stage":"app","5_msg":"Application myapp v4.1.0-alpha.1 launched"}
{"0_ts":"2024-10-24 09:30:12.330 +05:00","1_level":"debug",                                                                              "5_msg":"inotify test passed in 244.925µs"}
{"0_ts":"2024-10-24 09:30:12.313 +05:00","1_level":"error",  "3_stage":"licenseManager","4_command":"checkLicense",                      "5_msg":"product not configured"}
{"0_ts":"2024-10-24 09:30:27.438 +05:00","1_level":"warning","3_stage":"server",                                                         "5_msg":"Failed authentication attempt"}

These are not multiline and aren't used on live logs but they are often used with larger logs and trigger problems that I initially mentioned here.