[Bug] [Crash][Reproducible] EOFError: Ran out of input when import yapf with multiprocess

[whlook]

previous similar issue #1164

Problems

• When i use yapf (just import it) in multiprocess program, it will crash with 'EOFError: Ran out of inpu'，log info below:

Reproduction

1. yapf==0.40.2
2. rm -r ~/.cache/YAPF
3. run this python code:

 import multiprocessing
 
 def proc():
     import yapf
 
 if __name__ == "__main__":
     pp=[]
     for i in range(100):
         p = multiprocessing.Process(target=proc)
         pp.append(p)
 
     for p in pp:
         p.start()
 
     for p in pp:
         p.join()
                                                   

Reason

the problem code is here:

yapf/third_party/yapf_third_party/_ylib2to3/pgen2/driver.py

Line 247 in c0908d0

g = grammar.Grammar()

When in multi process enviroment,some process will create grammar cache(when ~/.cache/YAPF not exists), before the process write to the cache file, other process will see the file exist and load it, but the cache file is empty right now , so EOFError will raise and the program crashed.

How to fix it?

• I think we need to add try here(

  yapf/third_party/yapf_third_party/_ylib2to3/pgen2/driver.py

  Line 247 in c0908d0

  g = grammar.Grammar()

  ) like we had add try when we create the cache.
• Or we can add a multiprocess.Lock here
• Or we can do python -c 'import yapf' before we start our multiprocess program.(for users,trick)