PailRecordWriter using multiple OpenAttributeFiles fails to clean up

[derrickburns]

If a pail with multiple attributes is saved to S3, we have a problem.

For a pail with multiple attributes, multiple files are written. Unfortunately, these files are not updated atomically. Each AttributeFile is published by a rename operation independently. Therefore, if an update to a Pail with multiple attributes fails, some attribute files may be left on the file system.

When the exception is thrown, Hadoop will retry the operation. But it will simply fail again. Eventually, Hadoop will give up.

Further, simply deleting the files on the start of the job will not always work under S3, because deletes are only eventually consistent.