You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, the preprocessor splits the input file into multiple parts (using split). This part runs on a single core, because the splitting in its current form cannot be parallelized.
Modify the splitter to run on multiple cores:
Open N files, where N is the number of cores
Start N subprocesses to read from the input file
Each subprocess reads the input file entirely
nth subprocess only writes lines where line_number % N == N
The text was updated successfully, but these errors were encountered:
Currently, the preprocessor splits the input file into multiple parts (using split). This part runs on a single core, because the splitting in its current form cannot be parallelized.
Modify the splitter to run on multiple cores:
line_number % N == N
The text was updated successfully, but these errors were encountered: