pyspark.BarrierTaskContext.barrier

BarrierTaskContext.barrier()

Sets a global barrier and waits until all tasks in this stage hit this barrier. Similar to MPI_Barrier function in MPI, this function blocks until all tasks in the same stage have reached this routine.

New in version 2.4.0.

Notes

This API is experimental

In a barrier stage, each task much have the same number of barrier() calls, in all possible code branches. Otherwise, you may get the job hanging or a SparkException after timeout.