[CO-RE] relocations are not emitted for tracepoint 'context' argument.

[bieganski]

SEC("tracepoint/syscalls/sys_exit_openat")
int tracepoint__syscalls__sys_exit_openat(struct syscall_trace_exit* ctx)
{
	bpf_trace_printk("AAAA %d", ctx->ret);
	return 0;
}

And corresponding llvm-objdump -rd output:

Disassembly of section tracepoint/syscalls/sys_exit_openat:

0000000000000000 <tracepoint__syscalls__sys_exit_openat>:
       0:	79 12 10 00 00 00 00 00	r2 = *(u64 *)(r1 + 16)
       1:	18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00	r1 = 0 ll
		0000000000000008:  R_BPF_64_64	.rodata.str1.1
       3:	85 00 00 00 06 00 00 00	call 6
       4:	b7 00 00 00 00 00 00 00	r0 = 0
       5:	95 00 00 00 00 00 00 00	exit

What is the reason that 16-byte offset is hardcoded in first instruction?
I suspect that it causes issues when I cross-compile for different kernel and different architecture, though I use proper vmlinux.h.

tail of ./opensnoop output on my armv7l target:

49: (15) if r7 == 0x0 goto pc+50
 R0=map_value(id=0,off=0,ks=4,vs=16,imm=0) R6=ctx(id=0,off=0,imm=0) R7_w=map_value(id=0,off=0,ks=4,vs=16,imm=0) R10=fp0 fp-8=00000000 fp-16=00000000 fp-24=00000000 fp-32=00000000 fp-40=00000000 fp-48=00000000 fp-56=00000000 fp-64=00000000 fp-72=00000000 fp-80=00000000 fp-88=00000000 fp-96=00000000 fp-104=00000000 fp-112=00000000 fp-120=00000000 fp-128=00000000 fp-136=00000000 fp-144=00000000 fp-152=00000000 fp-160=00000000 fp-168=00000000 fp-176=00000000 fp-184=00000000 fp-192=00000000 fp-200=00000000 fp-208=00000000 fp-216=00000000 fp-224=00000000 fp-232=00000000 fp-240=00000000 fp-248=00000000 fp-256=00000000 fp-264=00000000 fp-272=00000000 fp-280=00000000 fp-288=00000000 fp-296=00000000 fp-304=00000000 fp-312=00000000 fp-344=mmmm????
; ret = ctx->ret;
50: <invalid CO-RE relocation>
failed to resolve CO-RE relocation <byte_off> [34] struct syscall_trace_exit.ret (0:2 @ offset 16)
processed 50 insns (limit 1000000) max_states_per_insn 0 total_states 1 peak_states 1 mark_read 1
-- END PROG LOAD LOG --
libbpf: prog 'tracepoint__syscalls__sys_exit_open': failed to load: -22
libbpf: failed to load object 'opensnoop_bpf'
libbpf: failed to load BPF skeleton 'opensnoop_bpf': -22
failed to load BPF object: -22

And corresponding llvm-objdump:

0000000000000000 <tracepoint__syscalls__sys_exit_open>:
       0:	bf 16 00 00 00 00 00 00	r6 = r1
       1:	b7 01 00 00 00 00 00 00	r1 = 0
       2:	7b 1a f8 ff 00 00 00 00	*(u64 *)(r10 - 8) = r1
       3:	7b 1a f0 ff 00 00 00 00	*(u64 *)(r10 - 16) = r1
...
      40:	7b 1a c8 fe 00 00 00 00	*(u64 *)(r10 - 312) = r1
      41:	85 00 00 00 0e 00 00 00	call 14
      42:	63 0a ac fe 00 00 00 00	*(u32 *)(r10 - 340) = r0
      43:	bf a2 00 00 00 00 00 00	r2 = r10
      44:	07 02 00 00 ac fe ff ff	r2 += -340
      45:	18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00	r1 = 0 ll
		0000000000000168:  R_BPF_64_64	start
      47:	85 00 00 00 01 00 00 00	call 1
      48:	bf 07 00 00 00 00 00 00	r7 = r0
      49:	15 07 32 00 00 00 00 00	if r7 == 0 goto +50 <LBB2_5>
      50:	79 68 10 00 00 00 00 00	r8 = *(u64 *)(r6 + 16)

[anakryiko]

You need very recent llvm-objdump to see CO-RE relocations with -r argument. If you have a bit older one, you won't see it, even though it's there. You can also try using https://github.com/anakryiko/btfdump to see recorded CO-RE relocations.

And also the way CO-RE relocations work is that compiler still emits a correct instruction (as of vmlinux.h definitions) with hard-coded offsets and so on, but also CO-RE relocation information pointing to this instruction. libbpf will then update instruction with adjusted offset, according to actual vmlinux BTF on target host.

You can also see all CO-RE relocations that libbpf is processing by looking at debug-level log output from libbpf.

[bieganski]

@anakryiko thanks for quick response.

I double checked and you are correct, newer llvm-objdump-18 shows CO-RE relocations properly.

The cross-platform BPF load failure was caused by 'fail_memsz_adjust', since it was trying to relocate *(u64*) load on 32-bit kernel.

Speaking of, do you think that it would make sense to introduce some kind of a unsafe_relaxed relocation resolving mode, so that size mismatch is allowed even on BTF_INT_SIGNED fields?

The rationale is that I inserted goto done just before res->fail_memsz_adjust = true; in https://github.com/libbpf/libbpf/blob/master/src/relo_core.c#L940 , and ended up with opensnoop and execsnoop examples working fine on 32-bit kernel, despite the fact that both are compiled on 64-bit kernel and both use *(u64*) load more than once.

Also, thanks for your help in resolving the issue!

[anakryiko]

libbpf doesn't fail BPF program loading, if relocation failed (but if it does, please provide full libbpf log, including debug level information). It just "poisons" those instructions, which is what will be rejected by verifier in the kernel. But, you can guard such poisoned instructions with additional checks, and if verifier sees that such poisoned code is unreachable, it won't complain. This allows to guard optional parts of code that rely on some unsupported (on old kernels) features, as long as it's properly detected and guarded.

[bieganski]

indeed. consider however such scenario: instead of

		if (res->fail_memsz_adjust) {
			pr_warn("prog '%s': relo #%d: insn #%d (LDX/ST/STX) accesses field incorrectly. "
				"Make sure you are accessing pointers, unsigned integers, or fields of matching type and size.\n",
				prog_name, relo_idx, insn_idx);
			goto poison;
		}
		orig_val = insn->off;
		insn->off = new_val;

we have this:

		if (res->fail_memsz_adjust) {
			pr_warn("prog '%s': relo #%d: insn #%d (LDX/ST/STX) accesses field incorrectly. "
				"Make sure you are accessing pointers, unsigned integers, or fields of matching type and size.\n",
				prog_name, relo_idx, insn_idx);
		       if !(config_allow_unsafe_relocs)
		           goto poison;
		}
		orig_val = insn->off;
		insn->off = new_val;

in that case user will be warned, and, for example, their 64-bit memory load will be clamped to lower 32-bits. and there is some chance that their code might still work properly (to some extent! but still might be useful).

so, if I understand correctly, you are saying about different code paths for different kernel version. and my use case is same code executed by any kernel, but in "unsafe" way, that might change it's semantics - by reducing width of memory accesses, if cross compiling on 64-bit kernel to 32-bit kernel. Note that we already do this clamping, but only for bool and char, not for arbitrary signed integer.

[anakryiko]

I don't think "sloppy_relocation" option is a good way forward. If you are running into this problem with a valid program, let's see if we can fix the problem itself. I'm going on vacation soon, so unlikely I'll have time for this in the next 2 weeks or so, but please share (minimal) source code and compiled .bpf.o file for your use case that fails. Thanks!