How to change the python version in Kaggle Notebook

Before we start changing the Kaggle Notebook Environment, We need to know how the Kaggle environment really works:-

The Notebook Environment

In Kaggle, Notebooks is more than just a code editor. It’s a versioned computational environment designed to make it easy to reproduce data science work. In the Notebooks IDE, you have access to an interactive session running in a Docker container with pre-installed packages, the ability to mount versioned data sources, customizable compute resources like GPUs, and more.

Notebook Versions and Containers

When you create a Notebook version using 'Save & Run All', you execute the Notebook from top to bottom in a separate session from your interactive session. Once it finishes, you will have generated a new Notebook version. A Notebook version is a snapshot of your work including your compiled code, log files, output files, data sources, and more. The latest Notebook version of your Notebook is what is shown to users in the Notebook viewer.

Every Notebook version you create is associated with a specific Docker image version as well. Docker is a containerization technology which provides an isolated environment in which to do your work. Docker specifies the contents of this environment including installed Python and R packages using what is known as an image. Every Notebook version you create is associated with a Docker image.

By default for new notebooks, this will be the latest version of the default Python or R images that we maintain at Kaggle. The contents of this image is publicly available on GitHub. You may view it at Github-docker-R for the R container, or Github-docker-python for the Python container.

Dockerfiles and Notebook Versions

Even if you are using one of the default Kaggle containers, the number, names, and versions of the packages that you’re using are still a moving target as our team continually updates them to ensure the latest and greatest packages are available. We update the images about every two weeks, mainly to upgrade to the latest versions of the packages we provide but also occasionally to add or remove certain packages. You can subscribe to notifications when we release a new Docker image on GitHub.

It is also possible to pin a specific Docker image for use in a Notebook if there are multiple custom images available. This can be done by accessing the “Settings” tab in the Notebook editor. Next to "Environment", there is an option to select "Preferences." This opens a modal where you can select what your environment preference is between pinning to a specific image (the image version when your notebook was created) or always using the latest image. You can read more about these options here.

In order to ensure that your Notebooks remain reproducible, we publicly expose the Dockerfile defining the environment the Notebook version was created in. You may download the contents of that Dockerfile by visiting the "Execution Info” section on your Notebook and navigating to the “Container image” field.

Steps to Change Python Version In Kaggle Notebook

In Kaggle Notebook the default python comes from the base conda environment, we can verify the conda version using the command,

which python
>>> /opt/conda/python

So in order to change the base python version in Kaggle, we have to change the conda base environment python version. There is a catch as well in changing the python version in base environment. If you just run the command

conda install python==3.6

This will probably run you into an error because the default base conda environment use the google cloud compute engine with the library dl-env-tf-version, finding the proper version matching with the correct python version is really time consuming. So instead of doing that, you can create a new environment with the required python version by running the command.

• Create a new conda Environment

conda create -n env-name python=3.6 -y

• Activate the enviroment

conda activate env-name

If you want to check the available conda environment, you can use the command

conda env list

• Check the default python path

which python

Also if you want to check the python version of your new conda environment

/opt/conda/envs/recitation/bin/python --version

• Now delete the default python path with the path of your environment python path

sudo rm /opt/conda/bin/python

• Set the new environment python path as the default python path

sudo ln -sf /opt/conda/envs/env-name/bin/python /opt/conda/bin/python

• Now check the python version

python --version

Thus using the steps above you can change the python version in kaggle Notebook.

Note: This is a temporary way of changing the Kaggle notebook python version, Everytime you run the code you will have to repeat the same steps mentioned above, to change the default python version.

The only permanent way to change the python environment is to run the code in Google Cloud Compute Engine. The Kaggle Notebook environment as the documentation says that it is a docker container it can save the variables and files and also some environment specific things in a state.db file in your /kaggle/working folder, which basically acts as the data storage to save files and variables to load the docker from its previous state.

Thus be careful while you run the code in the upgraded or downgraded python version make sure you follow the previous steps everytime you restart the code and the docker environment with accelerators.