A simulation environment for billiards: PoolTool

[ekiefl]

Hello, as mentioned in the welcome discussion, I have been working on a developing pool/billiards RL agents using the pooltool simulation environment, which I am the creator of.

My progress can be found in the following README file: https://github.com/ekiefl/LightZero/blob/main/README.fork.md

If you have suggestions, comments, are interested in collaborating on this project, or interested in incorporating pooltool as a supported environment in LightZero, please don't hesitate to reach out.

[puyuan1996]

Dear ekiefl,

Hello! We are deeply grateful for sharing your invaluable work with us. The pooltool simulation environment you've created holds great potential as a research benchmark in the field of reinforcement learning. It not only offers scientific value but also encompasses an element of fun. We believe integrating it as a long-term expandable benchmark environment will significantly contribute to the advancement of our research endeavors.

Given that we are currently in the midst of the Chinese Lunar New Year celebration, our team members and partners are spending precious time with their families. Thus, we may not be able to immediately engage in detailed discussions regarding our collaboration. But don't worry, we're very interested in this opportunity and plan to discuss the specific details of incorporating the pooltool environment into the LightZero project with you in a week's time.

Once again, we thank you for your patience and for your willingness to share this outstanding achievement with the community. We look forward to the possibility of further collaboration in the near future.

Best regards.

[TuTuHuss]

LGTM, i think we can discuss the specific details of incorporating the pooltool environment into the LightZero project.

[ekiefl]

That's wonderful to hear @puyuan1996 and @TuTuHuss.

No rush, please enjoy your holidays.

[puyuan1996]

Hello ekiefl,

• Thank you for your interest in the LightZero project and for your adherence to the realistic physics in the pooltool project. Your project is not only engaging but also especially suitable as a scalable benchmark environment for assessing long-term planning capabilities. I would like to inquire if you currently have the availability to integrate pooltool into LightZero as a new benchmark environment.

• We plan to start with the integration of your simplified game "Sum To Three." I hope you can prepare the relevant code (including environment setup, test files, and detailed API documentation, etc.) and submit a Pull Request in the specified format, similar to this PR: feature(xcy): add Connect4 environment #63. We can discuss specific questions on the PR afterward.

• Moreover, I have some questions about the performance issues encountered when combining the SampledEfficientZero policy in the "Sum To Three" game. In the current setup, is the game fixed to 10 steps per round, where the player outputs an action based on the current state at each step, thereby obtaining an immediate reward of 0 or 1, effectively concatenating 10 single-step MDP environments together?

• Having taken a preliminary look at your configs, possible directions for improvement include:

  • Increasing num_simulations, as the action space is continuous, leading to a large MCTS search space. Increasing this number generally helps to improve performance.
  • The reason for poor long-term performance might be the influence of old target policies in the buffer, necessitating a reanalyze_ratio = 1 setting.
  • Setting game_segment_length=10, as a game has a maximum of 10 steps.
  • Trying to set ssl_loss_weight=0, as the tb records you provided seem to show that this loss hasn't been optimized well. This setting is generally used in environments with image inputs and might have a negative impact in environments with vector inputs.
  • Check the magnitude variations of the mean and standard deviation of the actions outputted by the Gaussian distribution policy.

• Could you provide complete TensorBoard screenshots for your two observation settings, including the data collection (collect), evaluation (eval), and training (train) sections? This will help us analyze the current training condition and make targeted improvements.

• I look forward to collaborating with you, and I appreciate your patience and enthusiasm.

Best wishes!

[ekiefl]

Your project is not only engaging but also especially suitable as a scalable benchmark environment for assessing long-term planning capabilities

That is very exciting to hear. I'm excited to collaborate.

I would like to inquire if you currently have the availability to integrate pooltool into LightZero as a new benchmark environment.

I am definitely able to prepare a PR, however progress might be a bit slow because I am currently in the midst of job applications/negotiations.

I'll spend some time thinking about how we can best integrate pooltool. I think there will be many important decisions in deciding what is pooltool code and what is lightzero code.

Moreover, I have some questions about the performance issues encountered when combining the SampledEfficientZero policy in the "Sum To Three" game. In the current setup, is the game fixed to 10 steps per round, where the player outputs an action based on the current state at each step, thereby obtaining an immediate reward of 0 or 1, effectively concatenating 10 single-step MDP environments together?

In the current setup, the game is fixed to 10 steps per round. The player outputs an action, and receives a reward of 0 or 1 based on the outcome. However, I want to clarify that the player's action also defines the next observation. For this reason, I'm not sure if this is considered 10 single-step MDP environments, or 1 multi-step MDP environment.

Could you provide complete TensorBoard screenshots for your two observation settings, including the data collection (collect), evaluation (eval), and training (train) sections? This will help us analyze the current training condition and make targeted improvements.

Will send in a followup reply.

[ekiefl]

Vector observation:

I made some unfortunately undocumented changes between these runs. Probably the most important difference is the reward function in the orange experiment--please allow me to explain. In the orange experiment, possible rewards are 0 (failure, didn't hit object ball), 0.1 (failure, hit object ball) and 1 (success). In the other experiments, possible rewards are 0 (failure) and 1 (success). However, since the reduced action space uses "cut angle" (described here), which always aims at the oject ball, a reward of 0.1 is almost always guaranteed. That's why the orange episode reward min stays near 1.

Image observation:

[puyuan1996]

• Hello. Thank you for your patience. Feel free to prioritize your current tasks at hand. We can delve into the details of integrating pooltool into the LightZero zoo after you have managed your ongoing matters and prepared the corresponding Pull Request.

• Regarding the issue of sampled efficientzero performance, the vector observation's gray line, as indicated by the tensorboard data and environmental characteristics you've provided, already demonstrates a moderate level of effectiveness. The main areas for improvement remain those previously mentioned points. When your time permits, we can proceed with testing these improvements and conduct a thorough analysis.

• Your enthusiasm and contributions to the field of billiards and MCTS+RL are greatly appreciated.