-
Notifications
You must be signed in to change notification settings - Fork 96
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tensor data not read correctly from NPY files #144
Comments
What happens when you save a tensor from TF Scala and try to read it in Numpy? I only tested writing using TF Scala and reading back into TF Scala. |
@Spiess I actually just tried your example and it worked fine for me with Python 3.7. Are you sure you pulled TF Scala |
Ah, sorry, I was using TF Scala 0.4.1 and Python 2.7, so it's entirely possible it works on master. |
No problem. In that case, and since I can't reproduce this anymore on master, I'll close this. Feel free to reopen if the issue persists. :) |
I can confirm that npy files are not being correctly read in 0.4.2-SNAPSHOT either. Attached is I load the (uncompressed) npy file and inspect it as follows: val x_test = Tensor.fromNPY[Float](Paths.get("iris_x.npy"))
print(x_test.summarize()) This results in
The last 15 or so rows of the file are incorrectly read in as zeros. |
I have run the debugger through this. The problem appears in this synchronized {
// TODO: May behave weirdly for direct byte buffers allocated on the Scala side.
val directBuffer = {
if (buffer.isDirect) {
buffer
} else {
val direct = ByteBuffer.allocateDirect(numBytes.toInt)
val bufferCopy = buffer.duplicate()
direct.put(bufferCopy.limit(numBytes.toInt).asInstanceOf[ByteBuffer]) // <<< This copy is suspect
direct
}
} Shout if I can assist more. |
@davidmweber thanks for testing this and digging a bit into it! I’ll try to resolve it once I’m back from traveling in about 3-4 days and so for now I’ll simply reopen the issue. :) |
I have some code that I can repurpose to test ByteBuffer -> tensor and npy -> tensor implementations. the npy -> tensor needs some pre-stored file somewhere but I think it will be a useful test case |
@davidmweber This would indeed be a useful test case. Could you share that using a PR so I use it to debug this issue? Thanks! |
When I create a NPY tensor file from Python and read it from Scala TensorFlow, the shape of the resulting Tensor is correct, but the content is all zeros.
For example, writing from Python:
Reading from Scala TensorFlow:
Output:
The output from Scala TensorFlow is the same when saving the Python tensor with
allow_pickle=False
. The tensor can be read without any problems from Python using Numpy.The text was updated successfully, but these errors were encountered: