TypeError: 'NoneType' object is not iterable

[davetapley]

Describe the issue:

No ParquetException is thrown if metadata is corrupt.

Minimal Complete Verifiable Example:

return ParquetFile("corrupt.parquet")

I think I'm getting corruption due to bad file locking elsewhere in my app.
I'll attach file if I can catch the failure.

Anything else we need to know?:

I suspect None is returned here:

fastparquet/fastparquet/cencoding.pyx

Lines 693 to 702 in f747fe6

	def from_buffer(buffer, name=None):
	cdef NumpyIO buf
	if isinstance(buffer, NumpyIO):
	buf = buffer
	else:
	buf = NumpyIO(buffer)
	cdef dict o = read_thrift(buf)
	if name is not None:
	return ThriftObject(name, o)
	return o

Then we try and get [4] without a None check here:

fastparquet/fastparquet/api.py

Lines 221 to 225 in f747fe6

	fmd = from_buffer(data, "FileMetaData")
	except Exception:
	raise ParquetException('Metadata parse failed: %s' % self.fn)
	# for rg in fmd.row_groups:
	for rg in fmd[4]:

Environment:

• Dask version: fastparquet == 2023.2.0
• Python version: 3.10.9
• Operating System: Ubuntu
• Install method (conda, pip, source): pip

[davetapley]

Happy for a PR?

I owe one for ⬇️ anyway

• Support pathlib Path inputs #365 (comment)

[martindurant]

It is true that the low-level thrift reader does not raise errors, just ends up with a set of objects that are empty/None or otherwise not initialised. It would be reasonable to make a few basic verification checks and raising an error as you say.