Documenting FPrime packet structure

[rmelick-muon]

As someone new to FPrime, I am struggling to understand the structure of packets that are sent between the flight system and ground system.

First, I have a basic question: How much of this packet structure depends on my custom application components, and their ports , events, and telemetry channels as defined in the xml file?

Mostly though, I am interested in how these different packets look as bytes. I see some amount of this documented in code comments in the fprime-gds repository, but would love to see it in the official FPrime documentation. It looks like by default there are 8 different kinds of packets?

• FW_PACKET_COMMAND
• FW_PACKET_TELEM
• FW_PACKET_LOG
• FW_PACKET_FILE
• FW_PACKET_PACKETIZED_TLM
• FW_PACKET_IDLE
• FW_PACKET_HAND
• FW_PACKET_UNKNOWN

E.x. pkt_decoder.py shows a structure that looks like

+-------------------+---------------------+------------ - - -
| ID (2 bytes) | Time Tag (11 bytes) | Data....
+-------------------+---------------------+------------ - - -

I would love to fully document each of these 8 types, as well as the FPrime framing protocol that lives on top of them too.

I see a decent number of open questions and discussions about packet structure, if we can clarify the structure here I'm happy to add this documentation to the User guide in the appropriate location. Some examples include #1228, #1436.

[SterlingPeet]

This is definitely an open topic, and one that deserves attention. In my lab, we have reverse engineered some of the packet handling components to create alternate frame formats for our systems. Some of this work influenced the core development team to separate out Framer and DeFramer components, which has made this type of work easier.

I would also be interested in contributing if I can, but that may be limited to comments and suggestions for how to go about it.

[JackNWhite]

If you are interested, it is actually easier than it looks to follow TM data from an arbitrary component to the output (down)stream, and TCs from the input (up)stream to the destination component. Disregarding file transfer and, possibly, other non-core cases, there are TCs coming up, and two forms of TM.

Based on the Ref demo application (and bearing in mind we're still on F' 2.1.0):

Channel telemetry (most people's housekeeping) goes through these components:
Source component -> TlmChan -> Framer -> ByteStreamDriverModel

Through ports (per link above):
Tlm->TlmRecv, PktSend->comIn, framedOut->send

Event telemetry:
Source component -> ActiveLogger -> Framer -> ByteStreamDriverModel
Ports:
Log->LogRecv, PktSend->comIn, framedOut->send

Telecommands:
ByteStreamDriverModel -> Deframer -> CommandDispatcher -> Destination Component
recv->framedIn, comOut->seqCmdBuff, compCmdSend->cmdDisp

Following these chains will give you the packet formats. For TM each component, some data or metadata is serialised and appended or prepended to the existing serial. For TC, it's the opposite (although the initial decoding process is a little more involved).

For example, for event telemetry, which is the easiest to follow:

In component ArbitraryComponent, you have an event with three parameters. Upon invocation of the Log port, the three arguments are serialised, and sent along with the Event ID, send timestamp, and severity, to the LogRecv port of the ActiveLogger component.

Serialised: Arg1, Arg2, Arg3

Here, this package of data is queued until the cyclic activity is called. It is then popped and the packet type, event ID, timestamp and severity are serialised. The prior serial buffer is appended:

Serialised:
Packet type (Event), Event ID, Timestamp, Severity, Arg1, Arg2, Arg3

This buffer is then sent, along with a context number (which AFAIK is always 0), through the PktSend port to the comIn port of the Framer component. Framer passes the data to an object derived from FramingProtocol (FprimeFraming by default), where this data has any packet prefixes and suffixes attached. In F', this is the packet starting word 0xDEADBEEF, the size of the data, then the prior serialised data, and finally a checksum.

Serialised:
0xDEADBEEF, Data size, Packet type (Event), Event ID, Timestamp, Severity, Arg1, Arg2, Arg3, Checksum

At this point the packet is done and sent through the framedOut port to the send port of the ByteStreamDriverModel. Perhaps there's an additional checksum after that, I don't know.

[LeStarch]

I'd love help with this documentation, although I am at a loss how to get you the information you need without just doing the documentation. Below is an attempt:

We have a packet representing data using the above types, and a frame which frames those packets with a start word, size, and checksum.

Packets are described in the module comments in these files: https://github.com/fprime-community/fprime-gds/tree/devel/src/fprime_gds/common/decoders

The framing protocol is described here: https://github.com/nasa/fprime/blob/master/Svc/FramingProtocol/docs/sdd.md

@JackNWhite's above comments show the startword (0xdeadbeef), size, and checksum from the framing layer, as well as the serialized packet format:

Frame:
0xDEADBEEF, Data size, ||| inner packet format ||| Checksum

Inner Packet Format:

Packet type (Event/command/event), ID (opcode, channel id, event id), Timestamp, (packet specific data)

[rmelick-muon]

I think both of these are really helpful, and give me some stuff to go on. I'd like to write this up into a page that lives in the user guide somewhere. I'm thinking it will explain the structure, clarifying which parts come from FPrime (e.x. the 0xDEADBEEF), and which depend on the parameters you define for your components. I'll try to cover all the of the packet types built into FPrime. I will try to include links to the codebase that show where the structure is controlled.

Where would you suggest this documentation file live?

[LeStarch]

There is a new folder:

docs/Design

Do a pull request adding a .md write-up there. We have started putting design documentation there.

[rmelick-muon]

I have started work on documenting this (draft PR in my forked repo here https://github.com/rmelick-muon/fprime/pull/1/files), but I have gotten stuck on the more complicated ones, and run into a few questions:

1. There were some classes that didn't seem to follow the patterns that others did (GroundInterfaceComponentImpl and AmpcsEvrLogPacket). Are those deprecated classes or still used?
2. It seems that FilePacket has an additional layer of complexity / format. While investigating it was not obvious when the FilePacket class is actually invoked to pass its data to the framer.
3. What's the best way to link to other documentation within these readmes? That way I can link to the sdd docs when they have useful info.

[LeStarch]

1. Yes, both GroundInterfaceComponentImpl and AmpcsEvrLogPacket are deprecated.
2. FilePackets represent the internal structure of file data being sent up and down. It is the inner frame, and is written directly to the Fw::Buffer being passed to the Framer. Each subclass represents different types of packets (e.g. start, data, end) representing what stage of file transmission
3. New documentation like this should go in the UsersGuide docs/UsersGuide or design sections docs/design. You can link to SDD but remember to link to them like an external site not like an internal document as they are outside the docs directory.

[akshatd]

Hi @rmelick-muon I have been stuck developing something that can ingest F' Packets, specifically when the Telemetry is sent using TlmPacketizer. It was really hard to find any documentation, and after a day of digging and manually analyzing packets, I think I have a complete idea of how TlmPacketizer does things. I figured I'd let you know since you have a PR that's trying to document everything.

  Packet format:
  https://nasa.github.io/fprime/UsersGuide/best/ground-interface.html
  https://github.com/nasa/fprime-gds/blob/devel/src/fprime_gds/common/communication/framing.py#L94
  START (LSB)
  |  F' StartWord  | Packet Size  | Packet  |   Checksum   |
  |----------------|--------------|---------|--------------|
  |    4 bytes     |   4 bytes    | N bytes |    4 bytes   |

  Where Packet is (somehow there are extra 2 bytes before the ID???):
  https://github.com/nasa/fprime-gds/blob/devel/src/fprime_gds/common/decoders/pkt_decoder.py#L11
  |   ??    | Packet Type | Packet ID(??) | Time Tag |   Data  |
  |---------|-------------|---------------|----------|---------|
  | 2 bytes |   2 bytes   |    2 bytes    | 11 bytes | N bytes |

  Where Time is:
  https://github.com/nasa/fprime-tools/blob/devel/src/fprime/common/models/serialize/time_type.py#L78
  https://nasa.github.io/fprime/v1.5/UsersGuide/api/python/fprime/html/modules/fprime/common/models/serialize/time_type.html
  |  2 bytes  |    1 byte    | 4 bytes |   4 bytes    |
  |-----------|--------------|---------|--------------|
  | Time Base | Time Context | Seconds | Microseconds |

I was not able to figure out why there were 2 extra bytes in front of the ID when theyre not supposed to be there, so if someone can help me figure that out, it would be great. For example, this is what I received

0xDE 0xAD 0xBE 0xEF 0x00 0x00 0x00 0x55 0x00 0x00 0x00 0x04 0x00 0x01 0x00 0x02 0x00 0x67 0x44 0x3F 0x6A 0x00 0x06 0xF0 0xDF 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x38 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x90 0x35 0xDC 0xE0 0xDE 0xAD 0xBE 0xEF 0x00 0x00 0x00 0x15 0x00 0x00 0x00 0x04 0x00 0x03 0x00 0x02 0x00 0x67 0x44 0x3F 0x6A 0x00 0x06 0xF0 0xBD 0x00 0x00 0x00 0x00 0x8C 0xA5 0x7A 0x2D

Where I should have 4(FW_PACKET_PACKETIZED_TLM) when I use bytes 9 and 10, but instead it is at bytes 11 and 12. There is also an extra 2 bytes of packet type field, which isnt documented anywhere???

Here take note that each packet will be sent according to the definition in Packets.xml in the topology directory, and needs to be processed accordingly.

I hope one of the maintainers is able to get this into the documentation. It feels really incomplete for our use case