SIGTRAP Android freeze #1982
Comments
|
We would definitely need a reproducible example and also a sense of the APIs which are used (Reanimated version, Skia animations, etc) |
|
Hi @wcandillon, @LeviWilliams, We also experience this crash in production. So much - unfortunately - that we received a warning from Google regarding 'Android vitals bad behavior'. Our app will become less discoverable and receive a warning at the store page if this crash is not resolved soon. We currently have no clue when/where specifically this crash occurs. Therefore we cannot provide a reproducible example at the moment. For our app, in the last 7 days 1.5k users experienced 1.7k crashes of the SIGTRAP type, originated from Two crashes occur, both indicated by SIGTRAP, both about 50% of the total occurrences; and Some more details100% foreground crashes, spreaded to usage over Android versions and devices. Versions used in the context of the above; With Skia 0.1.210: No significant crashes Hope this helps tracking down the issue. |
|
@laurens-lamberts Thank you for this precious data.And I hope that we can get this sorted out as soon as possible. I need to review things more carefully on my side but from The error seems to be Skia specific, I will investigate this a bit deeper and let you know if I find anything. |
|
Strangely enough, I cannot find any relevant change from Going forward, we will setup some of release program to check if we introduce such regressions to releases. We have a RN Skia client of approximately the same scale as you running @laurens-lamberts I suggest we do the following:
|
|
Thanks a lot @wcandillon, we're really happy with your support proposal on this issue. Due to the high impact when issues arise during deployment in the christmas / newyear period, we will postpone next releases to January. For the upcoming release of our project we upgraded to the following library versions (all latest); If any new versions of the above packages appear before our release, we will update to ensure having the latest of all. We always perform our releases phased, so as soon as we got insight in crash rates we will share them with you. This will likely be the end of January / beginning of Februari. For my information, where in the react-native-skia library do I find the reference to the internal Skia version number (like |
|
This is were you can find the Skia version used : https://github.com/Shopify/react-native-skia/blob/main/.gitmodules I will continue to investigate this a bit and also do the upgrade to m121 and we can tackle this more aggressively after the holidays. I think that we are lucky to have a Skia client that uses 113 at scale but with only non-deprecated API, that will give us a lot of information once/when they deploy 114 and above. |
|
Yes, that's great. Looking forward to hearing their experiences with later versions. |
|
Any updates on this/ways to resolve it @laurens-lamberts @wcandillon @LeviWilliams ? Anything we can do to help? We just had to downgrade Skia to 196 because of thousands of crashes in production due to this error. Would be awesome to be able to upgrade since we want to move on to RN 0.73 (which according to release notes are not fully supported until 213) 🙏 |
|
@espenjanson Yes anything that would help to reproduce the issue or more details on the conditions of the crash would be extremely useful. I'm surprised you are on We have been coordinating with @laurens-lamberts to find the root cause of the issue but without success yet. We have a large client who's running the latest version of Skia without any crashes (this same client had a large amount of crashes in production with |
|
@espenjanson could you send me a list of Skia APIs and components you are using? You can do it privately as well by email. |
|
@wcandillon thanks for quick response. We'd love to help in any way we can. Any chance you could provide the package.json (or at least parts of it, such as react-native and reanimated version and perhaps other libraries that could affect skia)? If you want to, we can send you a minimal functioning app project with our crashing package.json and all the components we have that use Skia. Can put together a zip or a repo, whatever works better for you. If needed, we can also provide more detailed stack traces from Google Play and Sentry. Will put the team on this immediately. Thanks a million for paying attention to this! |
|
@wcandillon We have exactly same issue with quite big number of crashes with exact same error messages that started to My guess it's not possible to create 100% reproducible example because crash is quite random, but it's probably related to Skia + Animations. I will try to some app that uses same features as our production app and that will run that animations in some forever loop and hope it will crash after some time. Also I will try to just mount and unmount our components very quickly in some forever loop. I will let you know if I will find something. |
|
@Nodonisko is this on the latest version? It looks like this may have been fixed after the latest Skia version upgrade. |
|
Hi @wcandillon, We are live on 1.1.0 of react-native-skia and still experience the crash. Is this already using the latest Skia version? |
|
I'm slowly formulating a plan to tackle this issue. Is there a sense of which screen the crash is happening? This would allow us to funnel the API/code that might be faulty. |
|
@wcandillon We are not at latest version we have like two months old version. We will it but it will take us another month to test it in production. In mean time we were quite lucky and one of our testers managed to catch crash on video. It's not much helpful but at least we know on which screen it's happening. Sadly it's screen full of Skia components and animations :D I will try to prepare some standalone app from that screen so we can try to reproduce it in more isolated env. Hope I will have this done today or tmrw. screen-20240527-115456.mp4 |
|
nice, is this happening in debug mode? is there more details maybe
when the crash happens?
…On Tue, May 28, 2024 at 1:57 PM Daniel Suchý ***@***.***> wrote:
@wcandillon We are not at latest version we have like two months old version. We will it but it will take us another month to test it in production.
In mean time we were quite lucky and one of our testers managed to catch crash on video. It's not much helpful but at least we know on which screen it's happening. Sadly it's screen full of Skia components and animations :D
I will try to prepare some standalone app from that screen so we can try to reproduce it in more isolated env. Hope I will have this done today or tmrw.
https://github.com/Shopify/react-native-skia/assets/5837757/6ff2b12f-9c2a-4af3-a1ff-75cd84957560
—
Reply to this email directly, view it on GitHub or unsubscribe.
You are receiving this email because you were mentioned.
Triage notifications on the go with GitHub Mobile for iOS or Android.
|
|
Sadly it's production version. |
|
So I spend most of yesterday trying to reproduce the issue. I created special version of our homescreen that runs animations in loop, mounting unmounting components etc. and let it run on two different Android devices for 30 minutes few times. I also tried both production and debug builds. So far I did not get single crash... I also noticed that there is one single crash in our GPlay console that is happening with nearly same signature I am not sure if it's related or it's some completely unrelated random crash with similar signature, but it's suspicious. Any ideas what we can try next? We can try to deploy some special version of Skia from @wcandillon if that could help in our next release, but I will take another month to roll this out to our users. I will also try to narrow in which version crash occurred first. |
|
So it seems that crash occurred after update from Also found this issue in Skia issue trackers https://issues.skia.org/issues/333423686 |
|
@Nodonisko Does https://issues.skia.org/issues/333423686 look like it could be related? It wasn't clear to me looking the bug report @Nodonisko @laurens-lamberts @LeviWilliams I would like to find some way to reproduce the issue (even in release mode) and/or pin down the scenario in which the error happen. I wrote this example that stresses the Skia APIs and I tested also in release mode: https://github.com/user-attachments/files/15758252/StressTest.zip Please let me know how I could update the test scenario to better match your circumstances. In #2396, we are experiencing a clear race condition which we are currently investigating and that might shed some light on what is happening. |
|
@wcandillon Error in https://issues.skia.org/issues/333423686 looks very similar to our error but not sure it's related. About stress test, I run it on my two devices in debug mode and no crash so far. I also went through Sentry data and it seems it crashes very often (not exclusively) when user goes to new screen, both when previous screen is unmounted (like tab change) or also if new screen is pushed into stack, which leads me to idea that this probably happens when some Skia component is mounted. |
|
We just released new version of app with 1.3.7 Skia version and this issue still persist. My colleague just got this crash when he did some hover gesture over our graph (Revolut style graph animation). |
|
Hello @wcandillon @Nodonisko ! Any updates here? We're still experiencing both issues on react-native-skia version 1.3.9. Any workaround or solution to help us mitigate these would be greatly appreciated, as we’re currently seeing hundreds of crashes related to both problems Could we consider downgrading to version 196/210? @espenjanson /@laurens-lamberts mentioned in this thread that it helped them, though I don’t see this as a sustainable long-term fix, even if it does work. (Also, @espenjanson / @laurens-lamberts , did it actually resolve the issues? Have you encountered any more problems since?) other relevant information: Skia API used:
These APIs were used on the latest release, the one where we started seeing the crashes. We created a new component that is used on several lists (FlashList, FlatList, SectionList). Each list has somewhere around 10-50 items
|
|
We are observing the same pair of crashes reported above. Like other commenters above, the crash is rare enough that we have not been able to reproduce it consistently (or at all?) during development. react-native-skia has been a joy to program with, but this crash is a bit of a show stopper as it is happening frequently enough to raise our crash rate above the "bad behaviour" threshold in Google Play. We have been running react-native-skia version 1.3.13 and the issue persists after rolling out an optimistic upgrade to 1.5.2. A downgrade to 0.1.210 (a crash-less version reported above) is not possible without considerable effort to replace our use of newer react-native-skia features like ParagraphBuilder. |
|
@tmgrask We are actually working on a fix for this at the moment. We found a serious threading issue on Android which we are working on. These crashes are hard to reproduce so I cannot guarantee 100%. However the indication that |
|
The |
|
Good to know, I will follow that threading PR.
Just to be clear: we have not rolled out out a version with Thanks for the quick update and for the work on this great package! |
|
published 1.5.5 could you check to see if it helps? We fixed an important thread-safety issue there but these crashes can be hard to reproduce so I'd be curious to know if it helped. |
|
Thanks @wcandillon we will test it and let you know but I don't think we will have enough data earlier than end of December or early January due to our release process. |
|
still happening now even more, this was not happening to me before this update:
|
|
@joacub Any chance you would have an example that could help us reproduce the crash? |
|
expo: 52 return (
<Canvas style={{ flex: 1, position: 'absolute', left: 0, right: -1, top: 0, bottom: 0 }}>
<Rect x={0} y={0} width={width} height={height}>
<LinearGradientSkia start={start} end={end} colors={colors} />
</Rect>
</Canvas>
);downgrading skia to 1.5.4 works, last changes you did is what is causing this |
|
can you give me more of a snippet? what are width, height, start, end? |
that does not matter I already tested with the same snipped as you have in the demos and same behavior, you can the same example as skin docs has for linear gradient |
|
@joacub are you saying you are able to consistently reproduce the issue? You could help me reproduce it on my side? I would be also curious to know when it happens (e.g entering/leaving a particular screen for instance) |
yes it happen always in some android devices it is not related to the android version seems to be something with some devices, with one samsung galaxy a12 android 12 it is happening any time at the beginning just in the first render. skia version 1.5.4 is working, start happening in 1.5.5 |
|
Hi @wcandillon, I did some fuzz testing of our app using react-native-skia 1.5.4 and 1.5.6 in AWS device farm. Unfortunately it appears as though the SIGTRAP crashes remain, and 1.5.6 has introduced a new even more frequent crash (and faster, it prevents the first screen with skia on it from rendering at all). Here is what I see from device farm: Both versions have SIGTRAP crashes that originate in librnskia.so: 1.5.4: "Exerciser detected crash: Native crash: Trap (8)" 1.5.6: "Exerciser detected crash: Native crash: Trap (6)" Additionally, 1.5.6 has a new error that shows up quite frequently (more frequently than the SIGTRAP, at least in device farm): "Exerciser detected crash: java.lang.RuntimeException: Failed to create window surface (12)" (included that info log cause it always precedes the crash, and sounds bad) edit: I see this new error is also reported in #2754 |
|
I am working on subsequent improvements that I expect will improve the
situation. However, is there any chance you can show me the kind of
Skia usage is causing those crashes? As well as API version numbers.
It would be very useful if I could find ways to reproduce these
errors.
|
|
Our app uses many react-native-skia components and APIs, but there is really just 1 main screen where they are all combined, so I cannot pinpoint a specific one. Many of the react-native-skia components are animated via react-native-reanimated. From the videos of the fuzz testing, the crashes are not happening at the same time/with the same elements in view. I have also been trying to reproduce the issue in a controlled environment with no success. I will let you know if I stumble on any more relevant info. Through google play console I can see the SIGTRAP crashes are happening in producton on Android versions 8 through 14, so android API versions 28->34. I don't think we have enough data to say 35 is not also effected. |
|
v1.5.7 is published, I would love to get a read on whether it is improving Android stability? |
|
From early testing it looks like you have solved the new (#2754 ) crash in 1.5.7, nice work! Unfortunately, I still see the SIGTRAP crash in fuzz testing. I have been unable to find a way to reproduce locally but I'm still looking. |
|
Continuing to investigate the issue, #2761 will help a lot as well. I'm curious to see if it reduces the crashes. However if your app is crossing thread boundaries, you will now get null for some images in places where you might have gotten a result (but setting the direct context to a bad state). |
|
@wcandillon I am able to very inconsistently reproduce the crash in our app by running the Android exerciser monkey against a release build on an emulator. I almost hesitate to share this with you because I worry it will waste your time to chase this very rare repro. Our app source code is now public, I made a branch with the monkey script that (rarely) causes the crash: https://github.com/Psiphon-Inc/conduit/tree/skia-crash-repro The app uses react-native-skia in many places, but steps for very rare repro:
I just ran it 10 times and got it once, on the 7th time: I have yet to see it work from an Not exactly the most reproducible reproduction! |
|
Thank you @tmgrask this is huge. I'm am trying it now. On you side, if you use |
|
@tmgrask I was able to reproduce the crash but now trying to reproduce it with debug symbols and maybe adding some try/catches at the jni layer to see if some unhandled exception happens. I will publish 1.5.9 also which improve stability on Android. Any chance you could contribute the orb as a standalone example in https://github.com/Shopify/react-native-skia/tree/main/apps/paper/src? I didn't know about this monkey command but it's very useful :) I want to add it to the CI on both Skia and also RN WebGPU. Hope we can get to the bottom of this quickly. |
|
I tried Upgrading to 1.5.9 things look a little different: Here is a new crash I just saw locally (
When I put an APK with 1.5.9 through device farm fuzz test, there is another new crash that happens quite a bit. The backtrace doesn't explicitly mention rnskia, so perhaps a red herring, but it is noteworthy that I do not observe this crash in a build with 1.5.8. The number of SIGTRAP crashes is down (but still happening), however I have a feeling the SIGABRT crash above may be masking it, since the SIGABRT crash is happening relatively frequently. |
|
@tmgrask I ran the monkey script 10 times no crash, that's a good sign no? On previous version I feel like could reproduce it pretty quickly? |
|
@wcandillon 1.5.10 looks way better! Overall, device farm fuzz SIGTRAP rate is way down:
I am unsure if the SIGABRT EGL_BAD_ALLOC error is relevant, but it does reference I did repro SIGTRAP locally once, so it is still possible, but it does indeed seem much less common. This improvement is enough for me to ship an update to users, to hopefully alleviate our Google Play crash rate issue. SIGABRT (EGL_BAD_ALLOC):
Thanks! Your videos from beautiful Zurich Switzerland were very helpful! |
|
This is very exciting. Is it possible that the I love the monkey script, I really want to add it to the CI. if you want to contribute an example to the example app that would get tested there that would be fun. |
Description
Recently we upgraded our Skia version from 0.1.197 to 0.1.214 and now we are seeing a bunch of "SIGTRAP: Trace/breakpoint trap" in production on Android devices with a variety of versions. Currently on react-native 0.72.3 if helpful.
.
We reproduced a couple times on a Pixel 6 and the app just freezes. I understand this error is vague, we currently have no leads as to why this happens after updating the lib though if we are able to provide a repro we will.
Let us know if there are any ideas on what we can try, thanks for the help as always.
Version
0.1.214
Steps to reproduce
Snack, code example, screenshot, or link to a repository
The text was updated successfully, but these errors were encountered: