Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Installing Debug build MSI leads to ebpfsvc.exe not being able to start #3872

Open
lmb opened this issue Sep 26, 2024 · 22 comments · May be fixed by #3970
Open

Installing Debug build MSI leads to ebpfsvc.exe not being able to start #3872

lmb opened this issue Sep 26, 2024 · 22 comments · May be fixed by #3970
Assignees
Labels
bug Something isn't working P2 triaged Discussed in a triage meeting
Milestone

Comments

@lmb
Copy link
Collaborator

lmb commented Sep 26, 2024

Describe the bug

After installing from a Debug build MSI, the eBPF service fails to start with error 1053.

OS information

No response

Steps taken to reproduce bug

  1. Install Debug MSI via setup-ebpf.ps1
  2. Open services.msc
  3. Try to start the service

Expected behavior

The service should start.

Actual outcome

It doesn't.

Additional details

I think the problem lies in the fact that the service is configured to run as Local Service, which doesn't have access to the VC runtime debug dlls in system32. These are copied there by setup-ebpf.ps1. Copying the dlls into the JIT folder makes things work again.

@lmb lmb added the bug Something isn't working label Sep 26, 2024
@shankarseal shankarseal added triaged Discussed in a triage meeting P2 labels Sep 30, 2024
@shankarseal shankarseal added this to the 2410 milestone Sep 30, 2024
@shankarseal
Copy link
Collaborator

First, change the setup script to copy the DBG CRT locally. Second, please investigate if/why the service fails to load when the CRT is in system32 directory.

@vpidatala94
Copy link
Collaborator

attempting local reproduction

@vpidatala94
Copy link
Collaborator

setup script excepts the CRT DLL in the same location, thus copy action is a pre-step to the execution of this script hence not needed.

@vpidatala94
Copy link
Collaborator

Image

@vpidatala94
Copy link
Collaborator

Image

@vpidatala94
Copy link
Collaborator

Image

@vpidatala94
Copy link
Collaborator

Observation when setting - logon as local system account the issue is not seen - Image

@vpidatala94
Copy link
Collaborator

Seeing this error when using the installer to install ebpf with JIT - Image

@vpidatala94
Copy link
Collaborator

vpidatala94 commented Oct 16, 2024

Image - this shows that LOCAL SERVICE does not have read access to the debug dll "ending with d.dll", the reason being LOCAL SERVICE is part of Users group which is not granted read access for d.dll such as msvcp140d.dll in C:\Windows\System32 directory

@shpalani
Copy link
Collaborator

shpalani commented Oct 16, 2024

You need JIT installed for Debug msi, for running Kernel programs, along with vc_redist.x64.exe

@lmb
Copy link
Collaborator Author

lmb commented Oct 23, 2024

Could we copy the debug dlls to the JIT directory as a work around? (Side note: is the user expected to install the debug runtime when using an MSI without going via setup-ebpf.ps1?)

@lmb lmb removed the triaged Discussed in a triage meeting label Oct 27, 2024
@lmb
Copy link
Collaborator Author

lmb commented Oct 27, 2024

Removing the triage label because I'd like to discuss this in tomorrows meeting.

lmb added a commit to isovalent/ebpf-for-windows that referenced this issue Oct 27, 2024
Work around a permissions problem where the debug runtime DLLs in
System32 are not accessible to the ebpfsvc.exe service running as
Local Service.

Updates microsoft#3872
@shankarseal
Copy link
Collaborator

@vpidatala94 - please make changes to statically link CRT in debug bits.

@shankarseal shankarseal added the triaged Discussed in a triage meeting label Oct 28, 2024
github-merge-queue bot pushed a commit that referenced this issue Oct 28, 2024
Work around a permissions problem where the debug runtime DLLs in
System32 are not accessible to the ebpfsvc.exe service running as
Local Service.

Updates #3872
@vpidatala94
Copy link
Collaborator

Image
Image

@vpidatala94
Copy link
Collaborator

vpidatala94 commented Oct 30, 2024

this is after statically linking the debug runtime libraries to ebpf-for-windows.msi

@vpidatala94
Copy link
Collaborator

Image

@vpidatala94
Copy link
Collaborator

Image

@Alan-Jowett Alan-Jowett modified the milestones: 2410, 2411 Oct 30, 2024
@vpidatala94
Copy link
Collaborator

static linking will cause breakages in a scenario where - the customer is a using a release version of the product and debug version of the EBPF product. Fix addressed parallelly here - #3957, as part of this bug updating to documentation to help customers know that if they are using debug MSI to install ebpf directly they are supposed to copy the Debug DLL to a detectable location to allow it to be loaded

@vpidatala94 vpidatala94 linked a pull request Oct 31, 2024 that will close this issue
@lmb
Copy link
Collaborator Author

lmb commented Oct 31, 2024

@vpidatala94 if I understand you correctly you’ve tried running an application which is linked against the release runtime against an ebpfapi.dll statically linked against the debug runtime. Does this even work when dynamically linking the debug instead? I imagine that this might not be supported in the first place?

@lmb
Copy link
Collaborator Author

lmb commented Nov 1, 2024

The following makes me think that mixing release and debug versions isn't really supported in the first place:

All modules passed to a given invocation of the linker must have been compiled with the same run-time library compiler option (/MD, /MT, /LD).

So the use case of a customer using the release libs with debug ebpf for windows isn't really a valid use case? That would mean we can statically link after all.

@shankarseal shankarseal modified the milestones: 2411, 2501 Nov 2, 2024
@vpidatala94
Copy link
Collaborator

Apologies for the delay, the problem with static linking CRT is the ebpfapi.dll will have a different context compared to the Custom application (both using different sets of CRT) causing state difference/incompatibility as a result the ebpfapi.dll CRT will not be able to find the FD that customer application attempts to close. Based on discussion with Dave, the new direction we are going forth is making the debug installer more "complete" that the installer will will attempt to copy the DEBUG CRT to a set location included in path and if these are DEBUG CRT are not found, the installer will raise a warning.

@shankarseal
Copy link
Collaborator

@vpidatala94 please investigate if static linking against CRT works for all scenarios ( a C application loading ebpf api.dll etc.) If things turn out OK, when you change to static CRT, please revert #4014.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working P2 triaged Discussed in a triage meeting
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants