2021年2月23日火曜日

Jug, Parallel framework in Python

 Minor tips for problems I was into in Jug, a parallel computing framework in Python.
I may be wrong, and if you know I misunderstand, please leave comments:

  1. Tasks can be created only outside functions.
    Probably, this is common to create the main function with, if __name__ = __main__: main()
    I did this, and called functions decorated with @TaskGenerator. However, jug could not launch tasks that are created in the main function. Instead, when I remove the main function, and write lines on the ground level (no indent level), the same code worked.
  2. Multi-node parallelism can be achieved by just running "jug execute XXX.py" if the directory which stores the python code is shared by NFS. I could not find any clear comment on this, and I wondered how I can achieve multi-node parallelism.
  3. An atomic operation for File I/O can be achieved by creating a function decorated with @TaskGenerator. According to the summary of Jug, such a function is executed once. So, we can use it for an atomic operation.
Otherwise, I think Jug is excellent, and easy to use!

Still a question:
I do not know what jugfile can do.
On page17 of the documentation(https://jug.readthedocs.io/_/downloads/en/latest/pdf/), a post-process script import jugfile. I think the tasks in the main script make outputs to the jugfile, but do not know how to do.

0 件のコメント:

コメントを投稿