gen

Data Generation

We also provide code for generating PDDL-based expert demonstrations. This can be used to extend the training data, albiet without human language annotations.

Installation

Get dependencies and compile the planner:

$ sudo apt-get install ffmpeg flex bison

$ cd $ALFRED_ROOT/gen/ff_planner
$ make

Generation

To spawn multiple generation threads:

$ cd $ALFRED_ROOT/gen
$ python scripts/generate_trajectories.py --save_path data/new_trajs --in_parallel --debug --num_threads 2 

This will sample tasks based on the sampling mechanism described in the paper. You might notice a lot of failed executions, which are automatically discarded by the script.

Note: The first time you run the generation script, use --num_threads 1 to allow the script to download the THOR binary.

Replay Checks

In parallel with generation, replay saved trajectories to check if they are reproducable:

$ python scripts/replay_checks.py --data_path data/new_trajs --in_parallel  

This will ensure that the interaction masks and expert actions can be deterministically executed in THOR.

Data Augmentation

Currently, the dataset only provides 300x300 RGB images. However, each trajectory can be replayed to save any additional info available from the simulator. See the augment_trajectories.py script as an example for saving 600x600 RGB, depth and instance segmentation masks from the existing dataset:

python scripts/augment_trajectories.py --data_path data/json_2.1.0 --num_threads 2 --smooth_nav --time_delays

Note that these files consume a lot of storage space.

PDDL Tasks

The goals for the planner are specified in goal_library.py. Here is a simple pick-and-place PDDL goal definition:

# basic pick and place (e.g: "put the apple in the microwave")
gdict["pick_and_place_simple"] = '''
    (:goal
        (and
            ;; make sure all the cabinets and doors are closed in the end
            (forall (?re # receptacle)
                (not (opened ?re))
            )
            
            ;; make sure some object {obj} exists inside some receptacle {recep}
            (exists (?r # receptacle)
                (exists (?o # object)
                    (and 
                        (inReceptacle ?o ?r) 
                        (objectType ?o {obj}Type) 
                        (receptacleType ?r {recep}Type)
                    )
                )
            )
        )
    )
)