Upgrade to Clang 15 #7886

jez · 2024-05-13T23:33:02Z

Motivation

It's nice to be on more recent versions of our compiler.
In particular, it looks like it might have fixed a bug which means that
ClassOrModule is now 8 bytes smaller.
This will get us closer to being able to use the upstream
bazel-contrib/toolchains_llvm instead of our custom fork of
grailbio/bazel-toolchain. In particular, there seems to be something about
the way that the upstream invokes the linker such that our trick to get builds
working on macOS Sonoma again (bf58b62) does not work anymore.

Our current hypothesis for how to work around that is to switch away from
using /usr/bin/ld to link and instead to using the lld.ld from the LLVM
toolchain, which we think will give us a better ability to statically link
against libc++ on macOS (despite the OS's best effort to make that not
possible).
Upgrading toolchains_llvm is important because starting in Bazel 7 the
option --incompatible_enable_cc_toolchain_resolution will switch to
being on by default, and Sorbet's fork of bazel-toolchain uses the old
version. Also the upstream bazel-contrib/toolchains_llvm has better support
for building on Apple Silicon.

Test plan

Existing tests

I also plan to do some manual testing.

Required to make Clang 15 available.

Clang moved these at some point. Note that we only ever ran the sanitize build on x86_64--it never ran over arm64 builds. That might be nice to add some day, but I'm not blocking on it.

I couldn't get this silenced by adding `--copt=-Wno-array-parameter` because it looks like abseil builds this with a bunch of specific warning flags, and those take precedence over ours. The issue is fixed upstream, and when we upgrade we'll get this change for free, but as mentioned in the previous commit we can't upgrade abseil until after we upgrade clang.

In clang 15, the sanitizer was catching a stack-use-after-return error from this multithreaded codepath. The test which caught this was //test/cli:test_folder-input-not-found (Because it is multithreaded, the bug only manifested about 30% of the time, depending on the order that threads were scheduled by the OS.) The bug was that we were attempting to (atomically) write to `pendingJobs` from children threads after the function that spawned those threads exited. A sequence of steps that could happen: - A worker thread would realize that the input was not a directory. It would return a result requesting that the parent thread throw an exception, and queue up a bunch of jobs that would have the effect of shutting down the other threads. - The parent thread would get scheduled first. It would see that a child asked to throw, and it would throw, in effect returning from the function before all children were done. - The children would wake up and attempt to decrement the `pendingJobs` variable in the worker threads' closure by reference. That stack slot is not there anymore because the enclosing function raised. The fix is simply wait for all the threads to finish, and only raise the requested exception after we're sure that all the children threads have finished processing and shut down. For posterity, this is what the test failure looked like +================================================================= +==1781049==ERROR: AddressSanitizer: stack-use-after-return on address 0x7f435a57c140 at pc 0x55cac4f42fd8 bp 0x7f43420ef940 sp 0x7f43420ef938 +WRITE of size 8 at 0x7f435a57c140 thread T7 (options.findFil) + #0 0x55cac4f42fd7 in unsigned long std::__1::__cxx_atomic_fetch_add[abi:v15007]<unsigned long>(std::__1::__cxx_atomic_base_impl<unsigned long>*, unsigned long, std::__1::memory_order) realmain.cc + #1 0x55cac6e4d9b8 in std::__1::__atomic_base<unsigned long, true>::operator+=[abi:v15007](unsigned long) common.cc + #2 0x55cac6e3f5a6 in std::__1::__function::__func<appendFilesInDir(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, absl::flat_hash_set<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, absl::container_internal::StringHash, absl::container_internal::StringHashEq::Eq, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, sorbet::WorkerPool&, bool, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_0, std::__1::allocator<appendFilesInDir(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, absl::flat_hash_set<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, absl::container_internal::StringHash, absl::container_internal::StringHashEq::Eq, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, sorbet::WorkerPool&, bool, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_0>, void ()>::operator()() common.cc + #3 0x55cac629c0e1 in std::__1::__function::__value_func<void ()>::operator()[abi:v15007]() const driver.cc + #4 0x55cac6265778 in std::__1::function<void ()>::operator()() const (/pay/home/jez/.cache/bazel/_bazel_jez/2cc40ac946cd09178597ebd947795951/execroot/com_stripe_ruby_typer/bazel-out/k8-dbg/bin/main/sorbet+0x43e7778) (BuildId: c972efcad340ab8e0e5ea671063eb36b) + #5 0x55cac6ed26ac in std::__1::__function::__func<sorbet::WorkerPoolImpl::multiplexJob(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>)::$_2, std::__1::allocator<sorbet::WorkerPoolImpl::multiplexJob(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>)::$_2>, bool ()>::operator()() WorkerPoolImpl.cc + #6 0x55cac6ed12c5 in std::__1::__function::__func<sorbet::WorkerPoolImpl::WorkerPoolImpl(int, spdlog::logger&)::$_0, std::__1::allocator<sorbet::WorkerPoolImpl::WorkerPoolImpl(int, spdlog::logger&)::$_0>, void ()>::operator()() WorkerPoolImpl.cc + #7 0x55cac629c0e1 in std::__1::__function::__value_func<void ()>::operator()[abi:v15007]() const driver.cc + #8 0x55cac6265778 in std::__1::function<void ()>::operator()() const (/pay/home/jez/.cache/bazel/_bazel_jez/2cc40ac946cd09178597ebd947795951/execroot/com_stripe_ruby_typer/bazel-out/k8-dbg/bin/main/sorbet+0x43e7778) (BuildId: c972efcad340ab8e0e5ea671063eb36b) + #9 0x55cac6ee3e33 in Joinable::trampoline(void*) (/pay/home/jez/.cache/bazel/_bazel_jez/2cc40ac946cd09178597ebd947795951/execroot/com_stripe_ruby_typer/bazel-out/k8-dbg/bin/main/sorbet+0x5065e33) (BuildId: c972efcad340ab8e0e5ea671063eb36b) + #10 0x7f435c953608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x8608) (BuildId: c6d0d79d906d62bb768421fc6dada0d5e729f177) + #11 0x7f435c84c352 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x11f352) (BuildId: 87b331c034a6458c64ce09c03939e947212e18ce) + +Address 0x7f435a57c140 is located in stack of thread T0 at offset 320 in frame + #0 0x55cac6e3bb4c in appendFilesInDir(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, absl::flat_hash_set<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, absl::container_internal::StringHash, absl::container_internal::StringHashEq::Eq, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, sorbet::WorkerPool&, bool, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&) (/pay/home/jez/.cache/bazel/_bazel_jez/2cc40ac946cd09178597ebd947795951/execroot/com_stripe_ruby_typer/bazel-out/k8-dbg/bin/main/sorbet+0x4fbdb4c) (BuildId: c972efcad340ab8e0e5ea671063eb36b) + + This frame has 20 object(s): + [32, 40) '__p.addr.i.i.i.i' + [64, 65) '__af.i.i.i' + [80, 104) '__hold.i.i.i' + [144, 160) 'ref.tmp.i.i.i' + [176, 177) 'ref.tmp.i' + [192, 208) 'basePath' + [224, 225) 'recursive.addr' + [240, 244) 'numWorkers' + [256, 272) 'jobq' + [288, 304) 'resultq' + [320, 328) 'pendingJobs' <== Memory access at offset 320 is inside this variable + [352, 384) 'ref.tmp24' + [416, 464) 'agg.tmp41' + [496, 584) 'agg.tmp42' + [624, 656) 'threadResult' + [688, 704) 'ref.tmp73' + [720, 728) 'ref.tmp89' + [752, 832) '__enforceTimer' + [864, 880) 'agg.tmp199' + [896, 904) 'ref.tmp225' +HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork + (longjmp and C++ exceptions *are* supported) +SUMMARY: AddressSanitizer: stack-use-after-return realmain.cc in unsigned long std::__1::__cxx_atomic_fetch_add[abi:v15007]<unsigned long>(std::__1::__cxx_atomic_base_impl<unsigned long>*, unsigned long, std::__1::memory_order) +Shadow bytes around the buggy address: + 0x0fe8eb4a77d0: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 + 0x0fe8eb4a77e0: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 + 0x0fe8eb4a77f0: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 + 0x0fe8eb4a7800: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 + 0x0fe8eb4a7810: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 +=>0x0fe8eb4a7820: f5 f5 f5 f5 f5 f5 f5 f5[f5]f5 f5 f5 f5 f5 f5 f5 + 0x0fe8eb4a7830: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 + 0x0fe8eb4a7840: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 + 0x0fe8eb4a7850: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 + 0x0fe8eb4a7860: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 + 0x0fe8eb4a7870: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 +Shadow byte legend (one shadow byte represents 8 application bytes): + Addressable: 00 + Partially addressable: 01 02 03 04 05 06 07 + Heap left redzone: fa + Freed heap region: fd + Stack left redzone: f1 + Stack mid redzone: f2 + Stack right redzone: f3 + Stack after return: f5 + Stack use after scope: f8 + Global redzone: f9 + Global init order: f6 + Poisoned by user: f7 + Container overflow: fc + Array cookie: ac + Intra object redzone: bb + ASan internal: fe + Left alloca redzone: ca + Right alloca redzone: cb +Thread T7 (options.findFil) created by T0 here: + #0 0x55cac4eadd9c in pthread_create (/pay/home/jez/.cache/bazel/_bazel_jez/2cc40ac946cd09178597ebd947795951/execroot/com_stripe_ruby_typer/bazel-out/k8-dbg/bin/main/sorbet+0x302fd9c) (BuildId: c972efcad340ab8e0e5ea671063eb36b) + #1 0x55cac6ee4267 in runInAThread(std::__1::basic_string_view<char, std::__1::char_traits<char>>, std::__1::function<void ()>, std::__1::optional<int>) (/pay/home/jez/.cache/bazel/_bazel_jez/2cc40ac946cd09178597ebd947795951/execroot/com_stripe_ruby_typer/bazel-out/k8-dbg/bin/main/sorbet+0x5066267) (BuildId: c972efcad340ab8e0e5ea671063eb36b) + #2 0x55cac6ecf33f in sorbet::WorkerPoolImpl::WorkerPoolImpl(int, spdlog::logger&) (/pay/home/jez/.cache/bazel/_bazel_jez/2cc40ac946cd09178597ebd947795951/execroot/com_stripe_ruby_typer/bazel-out/k8-dbg/bin/main/sorbet+0x505133f) (BuildId: c972efcad340ab8e0e5ea671063eb36b) + #3 0x55cac6ece73c in sorbet::WorkerPool::create(int, spdlog::logger&) (/pay/home/jez/.cache/bazel/_bazel_jez/2cc40ac946cd09178597ebd947795951/execroot/com_stripe_ruby_typer/bazel-out/k8-dbg/bin/main/sorbet+0x505073c) (BuildId: c972efcad340ab8e0e5ea671063eb36b) + #4 0x55cac6a300a6 in sorbet::realmain::options::readOptions(sorbet::realmain::options::Options&, std::__1::vector<std::__1::unique_ptr<sorbet::pipeline::semantic_extension::SemanticExtension, std::__1::default_delete<sorbet::pipeline::semantic_extension::SemanticExtension>>, std::__1::allocator<std::__1::unique_ptr<sorbet::pipeline::semantic_extension::SemanticExtension, std::__1::default_delete<sorbet::pipeline::semantic_extension::SemanticExtension>>>>&, int, char**, std::__1::vector<sorbet::pipeline::semantic_extension::SemanticExtensionProvider*, std::__1::allocator<sorbet::pipeline::semantic_extension::SemanticExtensionProvider*>> const&, std::__1::shared_ptr<spdlog::logger>) (/pay/home/jez/.cache/bazel/_bazel_jez/2cc40ac946cd09178597ebd947795951/execroot/com_stripe_ruby_typer/bazel-out/k8-dbg/bin/main/sorbet+0x4bb20a6) (BuildId: c972efcad340ab8e0e5ea671063eb36b) + #5 0x55cac4f0379c in sorbet::realmain::realmain(int, char**) (/pay/home/jez/.cache/bazel/_bazel_jez/2cc40ac946cd09178597ebd947795951/execroot/com_stripe_ruby_typer/bazel-out/k8-dbg/bin/main/sorbet+0x308579c) (BuildId: c972efcad340ab8e0e5ea671063eb36b) + #6 0x55cac4eff162 in main (/pay/home/jez/.cache/bazel/_bazel_jez/2cc40ac946cd09178597ebd947795951/execroot/com_stripe_ruby_typer/bazel-out/k8-dbg/bin/main/sorbet+0x3081162) (BuildId: c972efcad340ab8e0e5ea671063eb36b) + #7 0x7f435c751082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082) (BuildId: 87b331c034a6458c64ce09c03939e947212e18ce) + +==1781049==ABORTING ================================================================================

froydnj

🎉

froydnj · 2024-05-16T13:52:27Z

cfg/CFG.cc

@@ -361,7 +361,7 @@ string BasicBlock::toString(const core::GlobalState &gs, const CFG &cfg) const {
 fmt::memory_buffer buf;
 fmt::format_to(std::back_inserter(buf), "block[id={}, rubyRegionId={}]({})\n", this->id, this->rubyRegionId,
 fmt::map_join(
- this->args, ", ", [&](const auto &arg) -> auto { return arg.toString(gs, cfg); }));
+ this->args, ", ", [&](const auto &arg) -> auto{ return arg.toString(gs, cfg); }));


These are very weird clang-format changes.

That's true. This is probably also the only place where we use an explicit -> auto return type in a lambda function in Sorbet?

jez requested a review from a team as a code owner May 13, 2024 23:33

jez requested review from neilparikh and removed request for a team May 13, 2024 23:33

jez force-pushed the jez-upgrade-zlib branch from ab6bc5f to 2ac0631 Compare May 15, 2024 17:51

jez force-pushed the jez-upgrade-clang branch from ee96c18 to 24d78d3 Compare May 15, 2024 18:06

Base automatically changed from jez-upgrade-zlib to master May 15, 2024 18:08

jez added 8 commits May 15, 2024 11:12

Pull in sorbet/bazel-toolchain#8

a78d5d6

Required to make Clang 15 available.

llvm toolchain: 12.0.0 → 15.0.7

c885b81

Silence this warning globally

d70465c

Fix "unused but set variable" warnings

ad62b2a

Stop linking against mimalloc on macOS release builds

751667f

Fix path to sanitizer archives

1d3ef77

Clang moved these at some point. Note that we only ever ran the sanitize build on x86_64--it never ran over arm64 builds. That might be nice to add some day, but I'm not blocking on it.

Add doc on how to build and publish LLVM

2ae1874

tools/scripts/format_cxx.sh

d5b8115

jez force-pushed the jez-upgrade-clang branch from cacff82 to 7d3488b Compare May 15, 2024 18:12

jez added 2 commits May 15, 2024 11:42

Silence some warning for abseil for now

e7f0852

jez force-pushed the jez-upgrade-clang branch from 7d3488b to c99938b Compare May 15, 2024 18:43

Ignore unknown warnings on emscripten build

ba7dd69

jez force-pushed the jez-upgrade-clang branch from c99938b to ba7dd69 Compare May 15, 2024 19:38

jez requested review from froydnj and removed request for neilparikh May 16, 2024 00:09

froydnj approved these changes May 16, 2024

View reviewed changes

jez merged commit 9d383a4 into master May 16, 2024
16 checks passed

jez deleted the jez-upgrade-clang branch May 16, 2024 17:31

vinistock mentioned this pull request May 27, 2024

Upgrade to LLVM 13 #5228

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade to Clang 15 #7886

Upgrade to Clang 15 #7886

jez commented May 13, 2024

froydnj left a comment

froydnj May 16, 2024

jez May 16, 2024

Upgrade to Clang 15 #7886

Upgrade to Clang 15 #7886

Conversation

jez commented May 13, 2024

Motivation

Test plan

froydnj left a comment

Choose a reason for hiding this comment

froydnj May 16, 2024

Choose a reason for hiding this comment

jez May 16, 2024

Choose a reason for hiding this comment