Self-hosted Docker images

Self-hosted Docker images can be useful for reducing the ingress costs, for accelerating image pulls, or for eliminating the dependency on Cortex's public container registry.
In this guide, we'll use ECR as the destination container registry. When an ECR repository resides in the same region as your Cortex cluster, there are no costs incurred when pulling images.
Step 1
Make sure you have the aws, docker and skopeo utilities installed.
Step 2
Export the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables in your current shell, or run aws configure. These credentials must have access to push to ECR.
Step 3
Choose a region for your cluster and ECR repositories. In this guide, we'll assume the region is us-west-2.
Also, take note of your AWS account ID. The account ID can be found in the My Account section of your AWS console.
Step 4
You can use the script below to push the Cortex images to your ECR registry. Make sure to update the ecr_region, aws_account_id, and cortex_version variables at the top of the file. Copy-paste the contents into a new file (e.g. ecr.sh), and then run chmod +x ecr.sh, followed by ./ecr.sh. It is recommended to run this from an EC2 instance in the same region as your ECR repository, since it will be much faster.
1
#!/bin/bash
2
set -euo pipefail
3
​
4
# user set variables
5
ecr_region="us-west-2"
6
aws_account_id="620970939130"  # example account ID
7
cortex_version="0.22.1"
8
​
9
source_registry="quay.io/cortexlabs"
10
destination_ecr_prefix="cortexlabs"
11
​
12
destination_registry="${aws_account_id}.dkr.ecr.${ecr_region}.amazonaws.com/${destination_ecr_prefix}"
13
aws ecr get-login-password --region $ecr_region | docker login --username AWS --password-stdin $destination_registry
14
​
15
# images for the cluster
16
cluster_images=(
17
  "manager"
18
  "request-monitor"
19
  "downloader"
20
  "operator"
21
  "cluster-autoscaler"
22
  "metrics-server"
23
  "inferentia"
24
  "neuron-rtd"
25
  "nvidia"
26
  "fluentd"
27
  "statsd"
28
  "istio-proxy"
29
  "istio-pilot"
30
)
31
​
32
# images for the APIs (you may delete any images that your APIs don't use)
33
api_images=(
34
  "python-predictor-cpu"
35
  "python-predictor-gpu"
36
  "python-predictor-inf"
37
  "tensorflow-serving-cpu"
38
  "tensorflow-serving-gpu"
39
  "tensorflow-serving-inf"
40
  "tensorflow-predictor"
41
  "onnx-predictor-cpu"
42
  "onnx-predictor-gpu"
43
  "python-predictor-cpu-slim"
44
  "python-predictor-gpu-slim"
45
  "python-predictor-inf-slim"
46
  "tensorflow-predictor-slim"
47
  "onnx-predictor-cpu-slim"
48
  "onnx-predictor-gpu-slim"
49
)
50
images=( "${cluster_images[@]}" "${api_images[@]}" )
51
​
52
extra_tags_for_slim_python_predictor=(
53
    "cuda10.0"
54
    "cuda10.1"
55
    "cuda10.2"
56
    "cuda11.0"
57
)
58
​
59
# create the image repositories
60
for image in "${images[@]}"; do
61
    aws ecr create-repository --repository-name=$destination_ecr_prefix/$image --region=$ecr_region || true
62
done
63
echo
64
​
65
# pull the images from Docker Hub and push them to ECR
66
for image in "${images[@]}"; do
67
    if [ "$image" = "python-predictor-gpu-slim" ]; then
68
        for extra_tag in "${extra_tags_for_slim_python_predictor[@]}"; do
69
            echo "copying $image:$cortex_version-$extra_tag from $source_registry to $destination_registry"
70
            skopeo copy --src-no-creds "docker://$source_registry/$image:$cortex_version-$extra_tag" "docker://$destination_registry/$image:$cortex_version-$extra_tag"
71
            echo
72
        done
73
    else
74
        echo "copying $image:$cortex_version from $source_registry to $destination_registry"
75
        skopeo copy --src-no-creds "docker://$source_registry/$image:$cortex_version" "docker://$destination_registry/$image:$cortex_version"
76
        echo
77
    fi
78
done
79
​
80
echo "###############################################"
81
echo
82
echo "add the following images to your cortex cluster configuration file (e.g. cluster.yaml):"
83
echo "-----------------------------------------------"
84
for cluster_image in "${cluster_images[@]}"; do
85
    cluster_image_underscores=${cluster_image//-/_}
86
    echo "image_$cluster_image_underscores: $destination_registry/$cluster_image:$cortex_version"
87
done
88
echo -e "-----------------------------------------------\n"
89
​
90
echo "use the following images in your API configuration files (e.g. cortex.yaml):"
91
echo "-----------------------------------------------"
92
for api_image in "${api_images[@]}"; do
93
    if [ "$api_image" = "python-predictor-gpu-slim" ]; then
94
        for extra_tag in "${extra_tags_for_slim_python_predictor[@]}"; do
95
            echo "$destination_registry/$api_image:$cortex_version-$extra_tag"
96
        done
97
    else
98
      echo "$destination_registry/$api_image:$cortex_version"
99
    fi
100
done
101
echo "-----------------------------------------------"
Copied!
The first list of images that were printed (the cluster images) can be directly copy-pasted in your cluster configuration file before spinning up your cluster.
The second list of images that were printed (the API images) can be used in your API configuration files. The image paths are specified in predictor.image (and predictor.tensorflow_serving_image for APIs with kind: tensorflow). Be advised that by default, the public images offered by Cortex are used for your predictors, so you will need to specify your ECR image paths for all of your APIs.
Step 5
Spin up your Cortex cluster using your updated cluster configuration file (e.g. cortex cluster up --config cluster.yaml).
Cleanup
You can delete your ECR images from the AWS ECR dashboard (set your region in the upper right corner). Make sure all of your Cortex clusters have been deleted before deleting any ECR images.

Guides - Previous

Set up kubectl

Next - Guides

Docker Hub rate limiting

Last modified 10mo ago

Copy link

Contents