Skip to content

korp_para

process_in_parallel(files_list)

Process file in parallel.

Source code in corpustools/korp_para.py 
12
13
14
15
16
17
18
19
def process_in_parallel(files_list):
    """Process file in parallel."""

    pool_size = multiprocessing.cpu_count() * 2
    pool = multiprocessing.Pool(processes=pool_size)
    pool.map(process_file, files_list)
    pool.close()  # no more tasks
    pool.join()  # wrap up current tasks