GitHub Code Search not returning results from some repositories

[fcrimmins]

Hi:

When I try a search for the term "Solr" in the Code Search preview, it returns a list of repositories where results were found. However, some repositories which contain the search term are not being returned in the list.

This is across our private repositories. I'm wondering if anyone else has come across this issue.

Thanks,

Francis.

[AndyAn7]

It's likely because they did not provide any topics while the code remains a minority in the total percentage of code used in the repo.

[colinwm]

Hey @fcrimmins, sorry about that. Were those repositories recently created? We currently have a bug where some recently-created repositories aren't being indexed, which we're working on fixing.

Also, can you be more specific about where the term appears in the code? Note that not all code is indexed by the code search preview, e.g. very large files (>300 kB) as well as some vendored or machine-generated code (like minified JS) aren't indexed.

[jonnytest1]

i have an example from
https://github.com/chromium/chromium/blob/main/chrome/browser/resources/inspect/inspect.js which contains "populateTargets" (line 1146)
but when i search for it in the repository i dont get any results

[fcrimmins]

Unfortunately the examples I have are all from private repositories for our organisation. The example repo was not recently created, and contains the term "Solr" across many different Java source files (as is it concerned with indexing content). None of these files is that large or machine generated. I would definitely expect this repo to be returned in results.

(Apologies for the delay in replying, I was on leave).

[sherimc]

Any updates on this bug? When a business has source code in GitHub and a developer has to search ALL of the source to analyze specific fields, we HAVE to have this functionality working and dependable. Thank you in advance and thank you fcrimmins for reporting this bug.

[Olshansk]

Is there an update on this?

For example, in Pocket OSS repo, searching for rancher turns up no results

But it is obviously present in this README:

Things I've tried:

1. Case sensitivity (doesn't work)
2. Wait for indexing (this code was merged in more than one month ago)

[colinwm]

Sorry about that, I'll try to trigger a reindex manually. But have you tried the new code search? https://github.com/features/code-search

That repo is indexed by the new search engine

[Olshansk]

That worked, ty!

[vanyakosmos]

same problem with backstage: https://github.com/backstage/backstage/search?q=gke&type=code

[look]

@vanyakosmos please try the new code search. The repo is indexed there. We are working hard to bring the new code search to 100% of users, but that does mean we don't have as much bandwidth to diagnose why repos aren't indexed in the older system.

[usetheplatform]

Hi. I experience a similar issue when searching code in private repositories, but I'm using GitHub REST API.
The search results from the API don't match the results I get from https://github.com/search.

[JP-Dhabolt]

I am also having similar issues. When I perform a search (using the new code search) across our org (internal repos), I get an incomplete result set.

For example: org:my-org "some-custom-search-term" path:my-file.yml returns 5 repositories (which I know is not all of them).

If I change to org:my-org "some-custom-search-term" path:my-file.yml repo:my-org/some-repo-that-did-not-show-up-in-previous-results I get a result for that repo, even though it was not included in the result set for the first search.

This drastically limits the utility of the new search functionality, as there is no confidence that it will actually return all of the results across a relatively limited set of repos.

[colinwm]

Are you searching for code or repositories? Can you confirm which tab you're looking at on the left pane here:

[JP-Dhabolt]

Searching for code. Code shows 1 (even though there are 5 results). Repositories show 0.

[juliencharpentier]

Hi ! I have the same problem with my organization, did you find a solution ?

[colinwm]

Hey, I think I know what the issue is, sorry about the confusing behaviour. The problem is that our search index deduplicates content by SHA, and this query was hitting some limits retrieving these documents since they were identical.

I'm going to try to fix this behaviour to avoid this since it is pretty obviously wrong, will keep you posted when the fix goes out so you can try it out again.

[Ayushi-gupta2001]

I also face some issues while searching particular keywords inside my organization repos. I am using Gihtub search APIs for this.

I have that particular keyword in my files in my org repo but not getting all of them. I am using this command for searching keywords.

curl -L -H "Accept: application/vnd.github+json" -H "Authorization: Bearer GITHUB_TOKEN " -H "X-GitHub-Api-Version: 2022-11-28" https://api.github.com/search/code?q=`search-keyword-term`.

This command is giving me incomplete results not expected results. Could anyone know is there are any changes that happened to GitHub search rest APIs?

[Ayushi-gupta2001]

yes, thanks for indexing. But If I have found the "fundwave" keyword in one of the repo, which is not in my organization but somewhere else, then this code search API is not giving that repo into my results.

I want to search some specific "keyword-name" worldwide on GitHub so do I need to index all repos in which it has that particular keyword?

like I have a file which is in my personal account: https://github.com/Ayushi-gupta2001/testing-branch/blob/main/fundwave.js

This file has that keyword, but when I am fetching this file through my code, it is not coming into results. Like all other files, which are not related to my organization, most of the files are not coming into results.

Could you please give me an answer to this, why has it happened?

[colinwm]

Hey @Ayushi-gupta2001, your repository Ayushi-gupta2001/testing-branch wasn't indexed by code search yet. If you search inside the repo (i.e. visit the repo page, and do a search from there), you'll trigger it for indexing.

I just did that, so it should be searchable within 10 minutes or so.

[ariefMudahMy]

@ariefMudahMy please give me the name of your org and I will reindex it.

hi @look sorry for late reply
org name is carousell

[colinwm]

@ariefMudahMy your organization's code should be indexed within ~10m or so

[ariefMudahMy]

@ariefMudahMy your organization's code should be indexed within ~10m or so

hi @colinwm thx

[Ayushi-gupta2001]

So do I need to index all the repository on GitHub to search code world wide.

…

On Tue, May 30, 2023, 11:26 PM colinwm ***@***.***> wrote: Hey @Ayushi-gupta2001 <https://github.com/Ayushi-gupta2001>, your repository Ayushi-gupta2001/testing-branch wasn't indexed by code search yet. If you search inside the repo (i.e. visit the repo page, and do a search from there), you'll trigger it for indexing. I just did that, so it should be searchable within 10 minutes or so. — Reply to this email directly, view it on GitHub <#18624 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANCQANSOGPHP6YGSBZA6KETXIYYFXANCNFSM5YZKNV3Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[look]

No, there are over 60 million repositories in the index and we are adding more every day.

[alexanderbkl]

It is incredibly awful that we can't sort by date the code we search on github...

[trevorflanagan]

I have same issue in one private organization that I am member of but not in another private organization. The one with the issue is a newly created organization (~1month). Is there anything that can be done to improve my search experience?

[jerry-b]

We are seeing this as a new issue in our repos as well. Searches that used to work are no longer returning results at all for full or partial matches on string. Previously we were only seeing failures when searching just for the numeric portion of strings such as "thisisathing-12345".

Note that this is NOT a new repository.

[look]

We are seeing this as a new issue in our repos as well. Searches that used to work are no longer returning results at all for full or partial matches on string. Previously we were only seeing failures when searching just for the numeric portion of strings such as "thisisathing-12345".

Searching for sequences of digits is a problem that we're aware of and working on. Limiting the search by org or repo(s) should help.

For the other searches, if possible, please provide example strings that don't work. Also let me know what the behavior is. Do you get no results, or do you see an error message?

Thanks.

[jerry-b]

Thanks for the response. I just tried to reproduce our issues with searching for digits, that seems to be resolved. The same searches that failed previously are now working. I found that the latest issue we had was looking for text in a source file that is 370K, so that is likely the remaining issue for us.

[look]

Thanks for checking in on it. If it happens again, please report it with as many details as you can so we can investigate. Appreciate it.

[franciscocha]

Is there a way for me to trigger the indexing? I'm trying to do a search in our private repositories but the search results are missing quite a bit of results.

[colinwm]

Can you elaborate a bit more on what's missing?

[franciscocha]

I searched for a term and get back 7 results in code. However I know that there are 18 files associated with the search term.

[con-gif]

@franciscocha at least you are getting 7 results. I am getting 0 results when there are at least 30+ files associated with the search term.

[franciscocha]

@con-gif I disagree. 0 results you know for fact that there is an issue. Partial results might actually make you take an action that you wouldn't have otherwise.

[franciscocha]

@look @colinwm is there an explanation for search not actually retrieving all of the results? Why is the search not returning all of the results found in code?

[peheje]

I have encountered the bug as well in my private repo which seems to impact the accuracy of search results. While this may seem a minor inconvenience in isolated circumstances, it has significant implications in a business context.

When we search for specific packages or endpoints within our organization's repository and find no results, we might inaccurately assume that it's safe to decommission them. Naturally, we utilize various techniques to inform our decisions, such as consulting with teams, analyzing logs, and more. However, the absence of any code related to a specific feature strongly suggests that it could be considered for decommissioning.

The consequences of such decisions could potentially disrupt our operations or development cycles, impacting the efficiency and productivity of our team.

Considering the potentially far-reaching effects of this issue, it would be helpful to hear from you about:

• Your official stance on this bug - acknowledging the issue and its potential ramifications.
• Identification of who might be affected by this bug.
• Steps that are being taken to rectify the issue.
• Expected timeline for resolving the bug.

Transparency and timely communication about such issues are crucial to help us manage our workflow, make informed decisions, and mitigate any potential risks.

I understand that bugs and errors are part of the iterative nature of software development. However, I believe that open and prompt communication about such issues is important in maintaining trust and providing excellent customer service.

Thank you for your attention to this matter. I look forward to hearing from you soon regarding this issue.

Kind Regards,

[jerry-b]

One technique that may help is to write a script that pulls a shallow clone of all repos, then search locally.

[grepsuzette]

If you search for kmscon, this repo https://github.com/Aetf/kmscon,
which is the most active one, does not appear.

[look]

@grepsuzette (great username, btw) can you post a separate feedback issue for this so we can route it to the correct team? This is related to repository search, not code search. I looked at it a little but couldn't figure out why the repository isn't showing up in the results. Thanks.

[Dr-Nutter]

Hi. I am seeing this same behaviour.

This search is only returning a single hit for a particular repo
https://github.com/search?q=repo%3Aedfenergy%2Fwmo-ma-cfd-cost-curve-uploader+-1.%23IND&type=code

and I know that there are multiple instances of the string "-1.#IND" in that repo.

I need to do a search across all of the repos in our organisation as this string needs to be updated in all of our tools.

Is it possible to get the repos in the edfenergy org indexed such that I can confident of the search results?

Thanks

[look]

I reindexed your organization's code in case that was the problem. However, it's also possible the results are being deduplicated, if the files in different repositories have the exact same content.

[Dr-Nutter]

Thanks for looking into this Luke.

Unfortunately, it has not resolved the issue. It seems that at least this one file is not being picked up in any searches. I've tried searching for other variable names and function names from this file and they are either not found in this file, if they are fairly generic variable names, or not found at all if they are very specific,

For example, the variable OUTPUT_DAILY_LOW_CFD_COST_CURVE (which is pretty specific to this one tool)

is not showing in an org-wide search

It is very likely that there are files in other repos in the org called main.bas, but the content of this file is definitely unique.

Thanks
Dave

[Yuri05]

However, it's also possible the results are being deduplicated, if the files in different repositories have the exact same content.

Is it possible to deactivate this behavior or provide a search option which turns the deduplication off? @look

[look]

@Dr-Nutter I looked into this, and this file is being excluded due to its content. Most commonly this means the file is too big (more than 350K) or is not valid UTF-8. Those limitations are documented. I'm interested in relaxing the limitations, but at this time the only work around is to keep your files below the size limit and ensure they are UTF-8.

@Yuri05 no, content-based deduplication is core to how the search engine works and cannot be turned off. We have heuristics to expand the results displayed when there are many duplicates. If you have examples or use cases where this isn't working for you, please let me know so I can file the feedback.

[Dr-Nutter]

Thanks Luke. Yes, the offending file is ANSI encoded. We'll go and look into why.

Cheers for digging into this.

[con-gif]

Sorry for writing again and notifying everyone again, but is there someone from the Github Team here ? Is someone working on this bug already or did they just give up ? I would really like to know if there will be a solution from Github otherwise I would try to think of an extern fix.

[samchoraria]

@colinwm
Can I request (re)indexing for org:ripjar please?

[look]

I've triggered a reindex for all of ripjar's repositories.

[jonnytest1]

Can I request automatic reindex for the top 10000 watched repos once a month ? ^^

[mbug]

@look Sorry, no action needed. I just realised that code search is just working if you are signed in.

[ShiraShuka]

I've encountered a similar issue, I've search for a term and got 30 results in 6 repositories. But the term actually appears in at least 3 more repositories, that I manually found.
when I've specifically named a repo it returned the results from it, but it's not reliable at all. That's a serious bug! I can't really know if i'm missing a usage.
[Private repo: Fiverr]

[reza-ebrahimi]

We made a private repo as public (with deleting private repo on github and creating new one and pushing data) and figured out that the old private repo's data coming up in the search! So dangerous! :)

How to trigger manually a repository to reindex?

[look]

Is your repository still showing up in search? It takes a few minutes to reflect changes in the search index.

[reza-ebrahimi]

@look I checked now and yes, still the deleted repo data showing in the search. Its now more than 3 hours.

[look]

Please file a support request (reference your post and direct them to @look) and I will investigate and correct the problem.

[reza-ebrahimi]

Thanks a lot @look, I already created a ticket at support (Ticket number: 2300545).
I just didn't get how to do direct to you on ticket, mentions not working there :)

[whenamanlies]

Is there a problem with specific file type/format perhaps?

We have https://github.com/dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/sabot/op/sort/external/MemoryRun.java

Search yields 0 results: https://github.com/search?q=repo%3Adremio%2Fdremio-oss%20MemoryRun&type=code

[look]

Sorry for the trouble. This is happening because our vendor/generated exclusion rules are being triggered due to this file being in a directory called external.

Thanks for raising the issue. We should revisit how we can tighten up these rules to avoid false positives like this.

[whenamanlies]

And what should/can people/companies do in the meantime?
What's the ETA? Is there an issue that could be tracked?

ps Thank you for the prompt reply!

[look]

We are aware of the issue and want to fix it, but there's no ETA (if we simply change the rules it could add quite a lot of content to the index and slow down searches). There isn't a public issue, but you could create a new discussion specifically for this if you like. I know this is unsatisfying, but if you rename the directory to almost anything else (like ext), the content will get indexed.

[lerchdude]

Hi, is resources also a folder name that will prevent indexing?
Example:
https://github.com/search?q=repo%3Adremio%2Fdremio-oss+%22Dremio+Free+Software+License%22&type=code
doesn't return
https://github.com/dremio/dremio-oss/blob/master/distribution/resources/src/main/resources-default/licenses/LICENSES_FOR_DEPENDENCIES.md#dremio-free-software-license-10

[pailhead]

Hi, I've been unable to find any code using search in 2023.

[advissor]

Same here. Github search, across my 150 repos used to be so effective! Loved it!
I was using it really often, especially to find some code piece to re-use

And now... I cant find anything. There are no matches found, very disappointed.
Really weird that this has not been raised enough

I can call the Github repo search - BROKEN

[nikszonszabo]

[jcarlosvale]

Hi, I am having the same problem, I am not able to find any code on my repos. Could you reindex it? I've tried common words and no answer.

[look]

Have you tried searching within the repo (repo:jcarlosvale/my-repo-name <term>)? That should trigger it to be indexed if it is not already.

[jcarlosvale]

I would like to search using all repos in my account and not one repo. Em qua., 18 de out. de 2023 19:16, Luke Francl ***@***.***> escreveu:

…

Have you tried searching within the repo (repo:jcarlosvale/my-repo-name <term>)? That should trigger it to be indexed if it is not already. — Reply to this email directly, view it on GitHub <#18624 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAI2NH2KBU3T7DSBVJWDWZTYAAMJPAVCNFSM5YZKNV32U5DIOJSWCZC7NNSXTOKENFZWG5LTONUW63SDN5WW2ZLOOQ5TOMZRHA3TEMY> . You are receiving this because you commented.Message ID: ***@***.***>

[yyz945947732]

Hi, I am having the same problem, I am not able to find any code on my repos.

example:

search:

https://github.com/search?type=wikis&q=%2F%2F+htmlLanguageParticipants+support.+Please+see+https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fvscode%2Fissues%2F160585+for+more+info.

cant find my code

my code:

https://github.com/yyz945947732/vscode-nue/blob/master/src/extension.js

[look]

So, if. I have 150+ repos ... I need to manually search in each of them to trigger indexing? This is so..so not right

Your repositories may already be indexed (for example, if they are actively used). We're working on displaying the indexed status of a repository in the UI to make this more clear.

I am not searching in VSCode, but github on the browser. I don't get any error or warning when searching within the repository where I know the word exists and it finds it within that repository.

If you don't get an error, your repository is in the index. If there are terms you can't find, you may be hitting some of our documented system limitations. For example, we don't index certain paths associated with vendored dependencies (e.g., node_modules), files larger than 350 KB, and files that are not valid UTF-8. If you can share the repository, I can look into it for you. As I noted above, we are working on UI to show when files aren't indexed.

[sherimc]

OK thank you for answering so quickly! I think that is the problem this program is over 1,000,000 lines long. Is there a way to know through GitHub how big it is?

[sherimc]

Never mind, I found it. That is the problem it is showing 725 kb.

[look]

Cool, I was going to suggest looking at the blob view. If you can split up the file into 3 files, you can get them all indexed. Otherwise, it's too big.

[sherimc]

Sadly that isn't an option as far as I know.

[adi-g15]

@look
Github code search not working in https://github.com/crash-utility/crash also. It's years old repository. Is the solution to re-index ?

Search String: "page excluded"
Search query: https://github.com/search?q=repo%3Acrash-utility%2Fcrash%20%22page%20excluded%22&type=code
The text in file: https://github.com/crash-utility/crash/blob/master/memory.c#L2295

[look]

@adi-g15 crash-utility/crash is indexed, but that file is too big to be indexed. It's over 500 KB and our limit is 350 KiB.

[adi-g15]

hi @look, does github enterprise also have this limitation ?

[look]

@adi-g15 GitHub Enterprise Server does not yet have the new code search engine (we are working on the changes necessary to support GHES). It does have a limit on file size, but it works differently: it truncates files rather than excluding them.

[adi-g15]

Oh, thanks for the information @look 👍

[kwitaszczyk]

@look Thanks for improving the code search. I've just noticed that we get "This repository's code has not been indexed yet. Try again later." when using the code search in CTSRD-CHERI/cheribsd and CTSRD-CHERI/cheribsd-ports (e.g., repo:CTSRD-CHERI/cheribsd sys_read, repo:CTSRD-CHERI/cheribsd-ports USE_PKG64) even though these repositories have been around for a long time. freebsd/freebsd-src and freebsd/freebsd-ports which these repositories fork and which are of a similar scale work fine.

Is there any way to ask GitHub to see if CTSRD-CHERI/cheribsd and CTSRD-CHERI/cheribsd-ports should be re-indexed?

[look]

CTSRD-CHERI/cheribsd was indexed when I checked and CTSRD-CHERI/cheribsd-ports was indexed on demand when I searched it. We had to disable on demand indexing for a while, but it has been re-enabled so if you see other repositories that are not indexed, please give them a search and wait ~5 minutes, then try again.