Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hyperparameter optimization #11

Open
danieleewww opened this issue Feb 14, 2016 · 1 comment
Open

Hyperparameter optimization #11

danieleewww opened this issue Feb 14, 2016 · 1 comment

Comments

@danieleewww
Copy link

Hello Mathis:

I ran through your solution using Intel® Core™ i5 CPU 760 @ 2.80GHz × 4 and GeForce GTX 970/PCIe/SSE2 under Ubuntu 14.04 LTS in about 18days, got kappa value shown in the attached which is similar as you did in the competition. I am planning either to go further with hyperparameter optimization or adjusting network depth or width. Any suggestion on this?

Cheers,

Daniel

screenshot from 2016-02-08 14 40 19

@sveitser
Copy link
Owner

I would probably try the following things, but I don't think there is much more to be gained this way either.

  • Make the networks deeper where possible.
  • Add another convnet to the ensemble.
  • Try using different rectifier units. Notably the elu unit that is now available in lasagne sounds promising.
  • You might be able to boost the score slightly if you try to compute optimal thresholds at the end rather than using 0.5, 1.5, 2.5, 3.5.
  • I think you can at least save some time during training if you remove the fully connected layers and/or add 'nonlinearity': None to the parameters for the output layer. The default linearity is rectify so the minimum the convnets can output at the moment is 0. You might have to tune the learning rate schedule again for this to work well though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants