Question 110
A Python UDF imports a helper library only on the driver, then executors fail with ModuleNotFoundError. Which change most directly addresses the operational problem?
Explanation
Correct Answer
Distribute the dependency through the cluster or job environment used by worker processes. This is the most directly relevant remediation because executor processes need access to the same importable module as the driver. A concrete acceptance check is: A distributed transformation runs on multiple partitions without import failures. To reduce recurrence: Test UDFs on worker executors rather than only local driver calls.
Incorrect Answers
Unconstrained transitive versions can introduce incompatible behavior between environments. Here the important technical distinction is that executor processes need access to the same importable module as the driver.
Paying for idle all-purpose compute is unnecessary for isolated short recurring tasks. The reported failure instead depends on this fact: executor processes need access to the same importable module as the driver.
Pipeline scheduling requires explicit data lineage to enforce upstream-before-downstream execution. In this situation, executor processes need access to the same importable module as the driver.