Patch 0.17 with commits from main. (#3604)

* Phase 1 of removing the rundown protection for invoke (#3534) * Phase 1 of removing the rundown protection for invoke Signed-off-by: Alan Jowett <[email protected]> * Fix SAL annotation error Signed-off-by: Alan Jowett <[email protected]> * PR feedback Signed-off-by: Alan Jowett <[email protected]> * PR feedback Signed-off-by: Alan Jowett <[email protected]> * Phase 1 of removing the rundown protection for invoke Signed-off-by: Alan Jowett <[email protected]> * Fix SAL annotation error Signed-off-by: Alan Jowett <[email protected]> * PR feedback Signed-off-by: Alan Jowett <[email protected]> * PR feedback Signed-off-by: Alan Jowett <[email protected]> * Ensure that _ebpf_program_free always run outside of an epoch to prevent deadlocks Signed-off-by: Alan Jowett <[email protected]> * Revert unrelated change Signed-off-by: Alan Jowett <[email protected]> * PR feedback and leak fix Signed-off-by: Alan Jowett <[email protected]> --------- Signed-off-by: Alan Jowett <[email protected]> Co-authored-by: Alan Jowett <[email protected]> * Cache choco packages to work around connectivity issues (#3536) * Cache choco packages to work around connectivity issues Signed-off-by: Alan Jowett <[email protected]> * Download file as cache key if needed Signed-off-by: Alan Jowett <[email protected]> * Skip choco if gather_dumps is false Signed-off-by: Alan Jowett <[email protected]> * Skip choco if gather_dumps is false Signed-off-by: Alan Jowett <[email protected]> * Apply suggestions from code review Co-authored-by: Dave Thaler <[email protected]> --------- Signed-off-by: Alan Jowett <[email protected]> Co-authored-by: Alan Jowett <[email protected]> Co-authored-by: Dave Thaler <[email protected]> * Add option to control the level of verbosity emitted by verifier. (#3570) * Add option to control the level of verbosity emitted by verifier. Signed-off-by: Alan Jowett <[email protected]> * Re-order enums to make normal < informational < verbose Signed-off-by: Alan Jowett <[email protected]> * Apply suggestions from code review Co-authored-by: Dave Thaler <[email protected]> * PR feedback Signed-off-by: Alan Jowett <[email protected]> * Workaround for verifier failure #643 Signed-off-by: Alan Jowett <[email protected]> * Revert change in ebpf-service verification path Signed-off-by: Alan Jowett <[email protected]> --------- Signed-off-by: Alan Jowett <[email protected]> Co-authored-by: Alan Jowett <[email protected]> Co-authored-by: Dave Thaler <[email protected]> * Add `total_size` to extension header. (#3573) * change ebpf_extension_header_t to ebpf_version_header_t * revert to ebpf_version_header_t * revert to ebpf_version_header_t * update header * update doc * update doc * use total_size to get the actual struct size * Apply suggestions from code review Co-authored-by: Dave Thaler <[email protected]> --------- Co-authored-by: Dave Thaler <[email protected]> * disable tailcall test for now (#3558) Co-authored-by: Dhiren Vispute <[email protected]> * Cleanup batch APIs and remove unused parameter (#3580) Signed-off-by: Alan Jowett <[email protected]> Co-authored-by: Alan Jowett <[email protected]> Co-authored-by: Sharmi <[email protected]> * Add support for multiple programs per section (#3557) * Allow multiple programs per section Temporarily use a fork of ebpf-verifier until vbpf/ebpf-verifier#642 is merged. Per https://stackoverflow.com/questions/13147170/attribute-always-inline-failing the __attribute__((always_inline)) doesn't do anything unless you also have the "inline" keyword. Signed-off-by: Dave Thaler <[email protected]> * Update verifier to latest Signed-off-by: Dave Thaler <[email protected]> * Address comment from Alan Signed-off-by: Dave Thaler <[email protected]> * Address API compat comment from Anurag Signed-off-by: Dave Thaler <[email protected]> --------- Signed-off-by: Dave Thaler <[email protected]> * Add workaround for NmrRegisterClient calling detach (#3594) Signed-off-by: Alan Jowett <[email protected]> Co-authored-by: Alan Jowett <[email protected]> * move enum to header (#3593) --------- Signed-off-by: Alan Jowett <[email protected]> Signed-off-by: Dave Thaler <[email protected]> Co-authored-by: Alan Jowett <[email protected]> Co-authored-by: Alan Jowett <[email protected]> Co-authored-by: Dave Thaler <[email protected]> Co-authored-by: Anurag Saxena <[email protected]> Co-authored-by: Dhiren Vispute <[email protected]> Co-authored-by: Dhiren Vispute <[email protected]> Co-authored-by: Sharmi <[email protected]>
microsoft · Jun 5, 2024 · 4167d2d · 4167d2d
1 parent 1be897b
commit 4167d2d
Show file tree

Hide file tree

Showing 162 changed files with 5,192 additions and 5,755 deletions.
diff --git a/.github/workflows/reusable-test.yml b/.github/workflows/reusable-test.yml
@@ -109,6 +109,42 @@ jobs:
       with:
         ref: ${{ github.event.workflow_run.head_branch }}
 
+    # Check if .github/workflows/reusable-test.yml exists locally.
+    - name: Check for .github/workflows/reusable-test.yml
+      # Check for test logs even if the workflow failed.
+      uses: andstor/file-existence-action@076e0072799f4942c8bc574a82233e1e4d13e9d6
+      if: (inputs.gather_dumps == true) && (steps.skip_check.outputs.should_skip != 'true')
+      id: check_reusable_test_locally
+      with:
+        files: .github/workflows/reusable-test.yml
+
+    # Check out just this file if code hasn't been checked out yet.
+    - uses: actions/checkout@0ad4b8fadaa221de15dcec353f45205ec38ea70b
+      if: (steps.check_reusable_test_locally.outputs.files_exists != 'true') && (steps.skip_check.outputs.should_skip != 'true')
+      with:
+        sparse-checkout: |
+          .github/workflows/reusable-test.yml
+        sparse-checkout-cone-mode: false
+
+    - name: Set up choco cache folder
+      # Set the choco cache to a local folder so that it can be cached.
+      if: (inputs.gather_dumps == true) && (steps.skip_check.outputs.should_skip != 'true')
+      id: choco-cache
+      run: |
+        mkdir ${{github.workspace}}\choco_cache
+        choco config set --name cacheLocation --value ${{github.workspace}}\choco_cache
+
+    - name: Cache choco packages
+      # Add cache entry for any choco packages that are installed.
+      # The cache key is based on the hash of this file so if any choco packages are added or removed, the cache will be invalidated.
+      if: (inputs.gather_dumps == true) && (steps.skip_check.outputs.should_skip != 'true')
+      uses: actions/cache@0c45773b623bea8c8e75f6c82b208c3cf94ea4f9
+      env:
+        cache-name: cache-choco-packages
+      with:
+        path: ${{github.workspace}}\choco_cache
+        key: ${{ hashFiles('.github/workflows/reusable-test.yml') }}
+
     - name: Install ProcDump
       id: install_procdump
       if: (inputs.gather_dumps == true) && (steps.skip_check.outputs.should_skip != 'true')

diff --git a/docs/debugging.md b/docs/debugging.md
@@ -81,10 +81,16 @@ r0 and r1, where r0 is for the return value and r1 holds
 the hook context (ctx) structure pointer that is
 passed to the program as its first argument.
 
-**Step 4)** To understand what went wrong, we can ask netsh for the verbose output by using "level=verbose":
+**Step 4)** To understand what went wrong, we can ask netsh for the informational or verbose output by using
+"level=informational" or "level=verbose":
+
+Note: Informational level will only show the first failure the verifier encounters on a specific path and not show
+dependent failures. Verbose level will show both the initial failures as well as failures arising from that initial
+failure. Informational is usually sufficient to understand the root cause of a failure, while verbose is useful to
+gain a deeper understanding of what the impact of this failure is.
 
 ```
-> netsh ebpf show ver droppacket_unsafe.o .text level=verbose
+> netsh ebpf show ver droppacket_unsafe.o .text level=informational
 Verification failed
 
 Verification report:
@@ -328,10 +334,10 @@ The destination of jumps are shown after the goto.  For example, instruction
 3 will jump to instruction 19 if the condition is true.
 
 
-**Step 7)** Let's now look at the verification failures of xdp_test using level=verbose:
+**Step 7)** Let's now look at the verification failures of xdp_test using level=informational:
 
 ```
-> netsh ebpf show ver droppacket_unsafe.o xdp_test level=verbose
+> netsh ebpf show ver droppacket_unsafe.o xdp_test level=informational
 ...
 Pre-invariant : [
     instruction_count=3,
@@ -599,7 +605,7 @@ which is passed as the map_key to `bpf_map_lookup_elem`.  Let's see how the veri
 understands this.
 
 ```
-> netsh ebpf show ver droppacket_unsafe.o xdp_test level=verbose
+> netsh ebpf show ver droppacket_unsafe.o xdp_test level=informational
 ...
 Pre-invariant : [
     instruction_count=28,

diff --git a/docs/eBpfExtensions.md b/docs/eBpfExtensions.md
@@ -64,13 +64,25 @@ initialized as follows:
 #### `ebpf_extension_header_t` Struct
 This is a mandatory header that is common to all data structures needed by eBPF extensions to register with the eBPF framework.
 * `version`: Version of the extension data structure.
-* `size`: Size of the extension data structure.
- When populating these data structures, the correct `version` and `size` fields must be set. The set of current version numbers and the
+* `size`: Size of the extension data structure, not including any padding.
+* `total_size` Total size of the extension data structure, including any padding.
+
+ When populating these data structures, the correct `version`, `size` and `total_size` fields must be set. The set of current version numbers and the
  size for the various extension structures are listed in `ebpf_windows.h`. For example:
 ```c
  #define EBPF_PROGRAM_TYPE_DESCRIPTOR_CURRENT_VERSION 1
  #define EBPF_PROGRAM_TYPE_DESCRIPTOR_CURRENT_VERSION_SIZE \
     (EBPF_OFFSET_OF(ebpf_program_type_descriptor_t, is_privileged) + sizeof(char))
+ #define EBPF_PROGRAM_TYPE_DESCRIPTOR_CURRENT_VERSION_TOTAL_SIZE sizeof(ebpf_program_type_descriptor_t)
+```
+
+When initializing the `ebpf_extension_header_t` struct, instead of using the individual values listed above, macros like below can also be used for convenience.
+```c
+#define EBPF_PROGRAM_TYPE_DESCRIPTOR_HEADER                                                              \
+    {                                                                                                    \
+        EBPF_PROGRAM_TYPE_DESCRIPTOR_CURRENT_VERSION, EBPF_PROGRAM_TYPE_DESCRIPTOR_CURRENT_VERSION_SIZE, \
+            EBPF_PROGRAM_TYPE_DESCRIPTOR_CURRENT_VERSION_TOTAL_SIZE                                      \
+    }
 ```
 > NOTE: Extension developers **must not** set the `size` field of these structures to `sizeof()` of the corresponding type. Instead,
 > the `CURRENT_VERSION_SIZE` macros defined in `ebpf_windows.h` should be used.
@@ -314,7 +326,6 @@ typedef ebpf_result_t (*ebpf_program_invoke_function_t)(
 /**
  * @brief Prepare the eBPF program for batch invocation.
  *
- * @param[in] extension_client_binding_context The context provided by the extension client when the binding was created.
  * @param[in] state_size The size of the state to be allocated, which should be greater than or equal to
  * sizeof(ebpf_execution_context_state_t).
  * @param[out] state The state to be used for batch invocation.
@@ -324,7 +335,7 @@ typedef ebpf_result_t (*ebpf_program_invoke_function_t)(
  * @retval EBPF_EXTENSION_FAILED_TO_LOAD if required extension is not loaded.
  */
 typedef ebpf_result_t (*ebpf_program_batch_begin_invoke_function_t)(
-    _In_ const void* extension_client_binding_context, size_t state_size, _Out_writes_(state_size) void* state);
+    size_t state_size, _Out_writes_(state_size) void* state);
 
 /**
  * @brief Invoke the eBPF program in batch mode.
@@ -345,13 +356,12 @@ typedef ebpf_result_t (*ebpf_program_batch_invoke_function_t)(
 /**
  * @brief Clean up the eBPF program after batch invocation.
  *
- * @param[in] extension_client_binding_context The context provided by the extension client when the binding was created.
  * @param[in,out] state The state to be used for batch invocation.
  *
  * @retval EBPF_SUCCESS.
  */
 typedef ebpf_result_t (*ebpf_program_batch_end_invoke_function_t)(
-    _In_ const void* extension_client_binding_context, _Inout_ void* state);
+    _Inout_ void* state);
 ```
 
 The function pointer can be obtained from the client dispatch table as follows:

diff --git a/ebpfapi/Source.def b/ebpfapi/Source.def
@@ -96,14 +96,19 @@ EXPORTS
     bpf_xdp_attach
     bpf_xdp_detach
     bpf_xdp_query_id
+    ebpf_api_elf_disassemble_program
     ebpf_api_elf_disassemble_section
+    ebpf_api_elf_verify_program_from_file
+    ebpf_api_elf_verify_program_from_memory
     ebpf_api_elf_verify_section_from_file
     ebpf_api_elf_verify_section_from_memory
     ebpf_api_close_handle
     ebpf_api_get_pinned_map_info
     ebpf_api_map_info_free
-    ebpf_enumerate_sections
-    ebpf_free_sections
+    ebpf_enumerate_programs
+    ebpf_enumerate_sections = ebpf_enumerate_programs
+    ebpf_free_programs
+    ebpf_free_sections = ebpf_free_programs
     ebpf_free_string
     ebpf_get_attach_type_name
     ebpf_get_next_pinned_program_path

diff --git a/external/ebpf-verifier b/external/ebpf-verifier