Import custom python modules

As we know “import” in python is to load python modules/methods to current python runtime.

For modules already imported in current python runtime, “dir()” can give a list:

# python3
>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__']
>>> sys
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'sys' is not defined
>>> import sys
>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'sys']
>>> sys
<module 'sys' (built-in)>

“import” looks up three places for requested modules: module cache for to see if it’s already imported, python standard library and other locations defined in sys.path.

In this note, we’ll use our YARN cluster lab env where Anaconda 2021.05 is deployed. We’re looking at sys.path in three different python runtime cases 1) conda python env, 2) python3 kernel and 3) pyspark kernel.

Lab ENV

Following diagram shows the three cases in which how sys.path looks like.

Case 1, sys.path in Conda Python ENV