Assert. FATAL 16897 handled. Core dump. Aborted.

[kitty99099]

Hello!
Based on this project #2889. I am going to use runhandler with rategroup 1Hz to send all of my telemetry (more than 50) so that all the data could be shown on the GDS. Although it can be really shown some data on the GDS for a while, it turned out to be core dump like below.

Assert: "/MyProject/build-fprime-automatic-aarch64-linux/Components/MyComponent/MyComponentComponentAc.c
FATAL 16897 handled.
Exiting with abort signal and core dump file.
Aborted

However, when I changed my number of telemetry sent a time to much less, the problem didn't happened or happened until a much longer time. I'm curious if this problem related to the buffer? I'm not sure how to find out the fundamental issue from these informations above. Thank you for your reading. I look forward to any useful response.

[thomas-bc]

fprime/config/FpConfig.h

Line 232 in dfc655f

#define FW_ASSERT_TEXT_SIZE 256 //!< Size of string used to store assert description

Making this config value larger in your config will allow the assert string to be longer, and we'll be able to see where the assertion is happening in the file, and therefore what is causing it.

[tbkevin]

I have more than 300 channels. I can't recall if that exactly error appeared to me, but for my case I had to increase "FW_COM_BUFFER_MAX_SIZE" found in "FpConfig.h", since this value will be check in the next assert, on an autogenerated cpp file:
"static_assert((700 <= (FW_COM_BUFFER_MAX_SIZE - Fw::Time::SERIALIZED_SIZE - sizeof(FwTlmPacketizeIdType) - sizeof(FwPacketDescriptorType))), "PacketsTooBig");"
That '700' value comes from your packet definition in your deployment, the xml file, 'NameOfYourDeploymentPackets.xml'. In my case I had:

Then i had to modify, in "TlmChanImplCfg.hpp", the next parameter:

Hope it helps you!

[kitty99099]

Hi @tbkevin , thank you for sharing your case.
After I figured out that I got the QUEUE_FULLproblem, I tried to survey and then found there is already a QA sililar to mine: #2825 .

In that QA @LeStarch said:

Increasing the active logger component's queue size or reducing the message/event rate is the only way to fix it.

In mine case, I can't reduce my message rate because of the design requirement. Therefore, I tried increasing my queue size. It seemed a bit helpful but not fully useful. When the queue was 100, it would trigger the assertion quickly. When the queue was 1000, it would trigger the assertion not until a longer time. When the queue was 10000, it would trigger the assertion not until a much longer time. All of them finally came out to be QUEUE_FULL .

I am curious that: when the queue get longer, shouldn't it be cleared automatically for a certain time just like buffer? I thought it could be fixed if I increase the queue size larger enough so that the queue can be cleared automatically. However, based on my result, it seems like I was wrong. Does anyone have insights?

***I was wrong! After more tests, the assertion time seems doesn't change even though I changed the queue size.
I changed the queue size in instance.fpp:

  instance bufferManager: Svc.BufferManager base id 0x4400 \
    queue size 10000 \
    stack size Default.STACK_SIZE \
    priority 100

So...I'm more curious now...

[LeStarch]

Queue overflows happen when a queue is not serviced fast enough to prevent overflow. This can be caused by:

1. Threaded component is receiving to many messages
2. Threaded component is processing too slowly
3. Queued component is not calling doDispatch() frequently enough

This assert is happening in MyComponent so you should be looking at the queue size of that component (no buffer manager as I see above).

[kitty99099]

Thank you! @LeStarch , I could'n find where to increase the queue size in the past few days. After your explaination, I found it in the instance.fpp as below.

  instance MyDeployment: Components.MyComponent base id 0x0F00 \
    queue size Default.QUEUE_SIZE \
    stack size Default.STACK_SIZE \
    priority 95

Then I tried to find what QUEUE_SIZE is. It is defined on the top of instance.fpp also.

  module Default {
    constant QUEUE_SIZE = 10
    constant STACK_SIZE = 64 * 1024
  }

I changed the QUEUE_SIZE from 10 to 100, and then it is better now.
Thank you for your guidence.