System packages

Cortex uses Docker images to deploy your models. These images can be replaced with custom images that you can augment with your system packages and libraries. You will need to push your custom images to a container registry that your cluster has access to (e.g. Docker Hub or AWS ECR).

Create a custom image

Create a Dockerfile to build your custom image:

mkdir my-api && cd my-api && touch Dockerfile

The Docker images used to deploy your models are listed below. Based on the Cortex Predictor and compute type specified in your API configuration, choose a Cortex image to use as the base for your custom Docker image.

Base Cortex images for model serving

  • Python (CPU): cortexlabs/python-serve:0.15.1

  • Python (GPU): cortexlabs/python-serve-gpu:0.15.1

  • TensorFlow (CPU or GPU): cortexlabs/tf-api:0.15.1

  • ONNX (CPU): cortexlabs/onnx-serve:0.15.1

  • ONNX (GPU): cortexlabs/onnx-serve-gpu:0.15.1

Note that the Docker image version must match your cluster version displayed in cortex version.

The sample Dockerfile below inherits from Cortex's Python CPU serving image and installs the tree system package.

# Dockerfile
FROM cortexlabs/python-serve:0.15.1
RUN apt-get update \
&& apt-get install -y tree \
&& apt-get clean && rm -rf /var/lib/apt/lists/*

Build and push to a container registry

Create a repository to store your image:

# We create a repository in ECR
export AWS_ACCESS_KEY_ID="***"
export AWS_SECRET_ACCESS_KEY="***"
eval $(aws ecr get-login --no-include-email --region us-west-2)
aws ecr create-repository --repository-name=org/my-api --region=us-west-2
# take note of repository url

Build the image based on your Dockerfile and push to its repository in ECR:

docker build . -t org/my-api:latest -t <repository_url>:latest
docker push <repository_url>:latest

Configure Cortex

Update your cluster configuration file to point to your image:

# cluster.yaml
# ...
image_python_serve: <repository_url>:latest
# ...

Update your cluster for the change to take effect:

cortex cluster update --config=cluster.yaml

Use system packages in workloads

Cortex will use your custom image to launch workloads and you will have access to any packages you added:

# predictor.py
import subprocess
class PythonPredictor:
def __init__(self, config):
subprocess.run(["tree"])
...