-
Notifications
You must be signed in to change notification settings - Fork 703
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
batch_size not fully implemented in DataSet class #1706
Comments
If you are not using equations you should try techniques suitable for purely data-driven surrogates. DeepXDE employs DeepONets for parametric learning. You could also try Fourier neural operators which is not available in DeepXDE. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello!
I am working on a model to predict neutron irradiation on materials, which basically feeds on large amounts of data, since using equations is not straightforward for this kind of predictions. The issue I encountered was that any training based on the dde.data.DataSet class would be really slow with my data, even when I changed batch_size to smaller values, but this wasn't the case with small datasets as the one in examples (dataset.train and dataset.test).
By further examining the Model class, I found that:
Original code found in DataSet:
Fix I used:
Finally, if I can make a suggestion, it would be helpful to have a mode for this DataSet class that loads the data in batches, instead of reading the whole dataset, since this can bloat the memory and crash the kernel.
I hope this helps, and thank you for the great work on this library, I am enjoying a lot learning from it!
The text was updated successfully, but these errors were encountered: