F Prime Data Products

[timcanham]

Introduction

F Prime currently supports the download of generated files via the Svc/FileDownlink component. The files can have any structure desired by users, and are opaque to the ground system. Developers are required to write their own file decoders to access the contents. In addition, users must add explicit download commands to sequences, which can have typos and fail due to unknows file sizes and telecom windows. A feature to improve this would be the concept of Data Products.

Data Products

Data Products would have the following features:

1. They would be structured. This means that they would have a typed representation described by the F Prime domain specific language (XML for the prototype, FPP for the final). This specification would allow the ground system and tools to decode them without manual implementation of tools. The ground system will provide a default text decoder, but would also allow users to override with a custom script. The data products will have a unique packet descriptor similar to other downlinked data like telemetry and events.

2. They would have multiple records. A data product could have multiple records in the generated file that are requested at run-time. This allows the storage of large generated data sets that whose size of may not be knowable until the activity is executed.

3. They would have a runtime specified priority. A generated data product would have a priority assigned to it that would allow the data product management system to prioritize the downlink of the set of data products currently on board.

4. The products will have containers. Containers are the objects that have a priority assigned. The container can hold any number of data product records of different types. This allows the collection of data to be managed as a unit for downloading and subsequent deletion.

5. The products and containers will be component-centric. Components will specify their own sets of containers and data products.

6. The containers and products will become new entries in the ground system dictionary.

Specification

The following is the proposed XML representation for data products:

<component name="UserComp" kind="active" namespace="SomeSpace">
   <import_serializable_type>SomeSpace/DataRecordSerializableAi.xml</import_serializable_type>
   <products>
      <container id="0" name="SomeContainer" default_priority="10"/>
      <container id="1" name="AnotherContainer"/>
      <record id = "0" name="IntRecord" type="U32" />
      <record id = "1" name="DataEntry" type="Somespace::Data" />
   </products>
</component>

Tag	attribute	Description
component		Starts specification of the component as currently specified in F Prime
import_serializable_type		Imports a serializable type to be used as a product record
products		Starts section specifying containers and products. Must be within the component specification
container		Specifies a container for data product records.
	id	Identifier for the container - offset from component ID
	name	The name of the container used for storing records. Used for ground system displays
	default_priority	Optional field to specify a default priority if not supplied at run time (> 0, 16-bit value)
record		A data product record that will hold data
	id	Identifier for the record - offset from component ID
	name	The name of the record. Used for ground system displays
	type	The type of the record. Can be basic F Prime types or a user defined serializable

Port Types

The following new ports will be defined for data products:

Fw/DpRequestPortAi.xml - output port from component to request a data product buffer

<interface name="DpRequest" namespace="Fw">
    <comment>
        Request that a Data Product buffer will be supplied for the requester
    </comment>
    <args>
        <arg name="ID" type="U32">
            <comment>ID for requested buffer. Matches container ID.</comment>
        </arg>
        <arg name="size" type="U32">
            <comment>size of requested buffer</comment>
        </arg>
    </arg>
</interface>

Fw/DpSupplyPortAi.xml - input port to component to supply a buffer for filling with data products

<interface name="DpSupply" namespace="Fw">
    <comment>
        Supplies a data product buffer
    </comment>
    <include_header>Fw/Buffer/Buffer.hpp</include_header>
    <args>
        <arg name="ID" type="U32">
            <comment>ID for requested buffer. Matches container ID</comment>
        </arg>
        <arg name="buffer" type="Fw::Buffer">
            <comment>supplied buffer</comment>
        </arg>
    </arg>
</interface>

Fw/DpWritePortAi.xml - output port to send filled buffer

<interface name="DpWrite" namespace="Fw">
    <comment>
        Sends filled data buffer
    </comment>
    <include_header>Fw/Buffer/Buffer.hpp</include_header>
    <args>
        <arg name="ID" type="U32">
            <comment>ID for requested buffer. Matches container ID</comment>
        </arg>
        <arg name="buffer" type="Fw::Buffer">
            <comment>filled buffer</comment>
        </arg>
    </arg>
</interface>

Component Functions

The code generator will supply the following base class functions:

Dp_Request(ContainerIdEnum id, U32 size)

This will request a buffer to fill with data product records.

The ContainerIdEnum type is a generated enumeration of the name IDs that are part of the container specification. For the above example, it would look like:

enum ContainerIdEnum {
    SomeContainer = 0,
    AnotherContainer = 1
};

The size argument should be the max size in bytes of the buffer. This would consist of multiplying the size of the individual record by the number of records that the user intends to store in the buffer. For serialized types, the size would be the SERIALIZED_SIZE constant in the serializable header file, while built-in types could use sizeof.

The autocoder will generate a virtual function:

Dp_Write_handler(ContainerIdEnum id, Fw::Buffer& buffer)

This will supply a copy of Fw::Buffer. The user can then serialize the number of records of the data structure to the buffer.

Dp_SerializeRecord_DataEntry(Fw::Buffer& buffer, const Somespace::Data& record);

This serializes a particular record. It is called as many times as necessary to fill the size requested.

void SomeComponent::Dp_Write_handler(ContainerIdEnum id, Fw::Buffer& buffer) {

    Somespace::Data someData;
    ...

    Dp_SerializeRecord_DataEntry(buffer, someData); // do this N times
    ...
    Dp_WriteProduct(id,buffer); // send product buffer to be written; see below

}

Dp_WriteProduct(ContainerIdEnum id, Fw::Buffer& buffer);

This function will send the buffer back to the component writing the DP.

New Components

Diagrams to come...

DpWriter

DpWriter will do the following:

1) Service requests for DP Buffers (DpRequest port)
2) Call Svc::BufferManager to request buffers
3) Call DpSupply port to send buffer to component
4) Receive the buffer filled with data products
5) Time-tag product generation (user will have to time-tag entries if needed)
6) Send the buffer out a Fw::BufferSend port. This can be connected to Svc::BufferAccumulator or Svc::BufferLogger depending on the use case. They are expected to send it back to the sourcing BufferManager when done.

DpCatalog

DpCatalog will do the following:

1) Upon initialization, read the DP directory and build a catalog of products
2) When commanded, start downloading based on priority and other TBD criterion (date? size?)
3) Manage deletion of data products (commands or file list)
4) Generate DP listings for download (file? self-defined DP?)

New Ground System work

1) Provide decoder for DPs based on new dictionary entries. Default would be text, but allow users to supply their own custom based on DP ID.
2) Provide download status views for GUI.
3) Provide view that shows current listings from DpCatalog listings.

Please add any comments below...

[gcgandhi]

A few features we may want to consider in the future (some discussed already):

1. DP re-transmission (should be able to do an entire DP or "parts" of DP)
2. Make CCSDS compliant
3. Auto-deletion of DPs
4. DP compression by command and/or auto-compression
5. RAM-caching for performance improvements (instead of writing directly to the file system)
6. "Continuous" DP support for recorded EHA/EVRs, GNC streaming logs, etc
7. Option for a-synch. DP operations to support DPs that are "Continuous" or have strict time requirements (i.e. DPs to debug anomalies).
8. Conversion tool to make "chill_dp_view" .xml compliant

[ThibFrgsGmz]

Personally, I have never experimented with F Prime components manipulating blobs (text, log, video, audio files... or even a file system) because my embedded software never has file systems. Who knows, maybe one day?

Your feature, on the other hand, is interesting and well thought out, and it seems to meet one of your internal needs. In that case, it would be interesting to implement what you describe, but I don't have enough expertise to judge it properly at the moment.

[bdshenker]

Given that container is of predefined size, consider adding attribute overflow_policy to tag container with at least 3 possible values: WRAP_ON_OVERFLOW, OVERWRITE_LAST_ON_OVERFLOW, DROP_ON_OVERFLOW.

[timcanham]

@bdshenker The containers aren't a predefined size. A size is requested at runtime. The user should programmatically know the size, but in the event of a mistake in sizing, the return values from serializing the records will tell the user when the container is full.

[timcanham]

Addtions:

Notion of quotas per DPID
"lossy" vs. "nonlossy", i.e. products that can't be lost vs. ones that could be dropped.

[timcanham]

Some updated discussions:

1. Specialized DpWriter component that would replace BufferLogger as the writer of data products. It would generate a file per container with a naming convention of <APID>_<time stamp>.fdp. It will also have data compression ports for optionally compressing data prior to writing it to storage.

2. The previously named DpWriter could become DpManager to more accurately represent its role as allocating memory for DPs and forwarding generated DPs to BufferAccumulator or DpWriter.

[timcanham]

Some more ideas:

Upon activation, the DpCatalog should (optionally, one of the activation command arguments) generate a catalog listing DP with container IDs, dates and sizes and make that the first DP downlinked.

Add the notion of Dp metadata in the container header - the user could have (two?) U32 fields with arbitrary values to inform the ground about the content (say, "flight1, image12"). That could be part of the catalog listing DP.

[timcanham]

Add some ports to DpCatalog that would:

1. allow another component to ask for the next Dp file in the catalog
2. allow another component to report completion transferring the Dp so it can be marked "done"

[bocchino]

The current design on the FSW side is documented here: https://github.com/bocchino/fprime/blob/data-products/docs/Design/data-products.md.

[timcanham]

Add a command to DpCatalog that will block until the DP catalog has been transferred. That can allow behaviors like powering down after completion.