Docker Hub rate limiting

Docker Hub's newly enforced rate-limiting policy can negatively impact your cluster. This is much likelier to be an issue if you've set subnet_visibility: private in your cluster configuration file, since with private subnets, all requests from all nodes are routed through the NAT Gateway, and will therefore have the same IP address (docker imposes the rate limit per IP address). If you haven't specified subnet_visibility or have set subnet_visibility: public, this is less likely to be an issue for you, since each instance will have its own IP address.
If you are affected by Docker Hub's rate limiting, your may encounter issues such as:
your APIs typically run as expected but new replicas (during scale up) or newly submitted batch jobs suddenly stop working for a period of time and then eventually they start working again
you encounter scaling issues for Realtime APIs
batch jobs are stuck in an in progress state for an unusually long period of time
Follow these steps to determine if this issue is affecting your cluster:
1.​Setup kubectl​
2.kubectl get pods --all-namespaces
3.Check the pod status column for image pull failures such as ErrImagePull, ImagePullBackoff. If you don't see any, the rate limiting may not be affecting you currently.
4.Get the pod id and namespace of a pod encountering image pull failures
5.kubectl describe pod <pod id> --namespace <pod namespace>
6.Under the events section, if you see error events related to docker hub rate limiting, then your cluster is likely affected by the rate limiting
There are three ways to avoid this issue:
Use Cortex images from quay.io
In response to Docker Hub's new image pull policy, we have migrated our images to quay.io. This registry allows for unlimited image pulls for unauthenticated users.
It is possible to configure Cortex to use the images from Quay instead of Docker Hub:
Update your cluster configuration file
Add the following to your cluster configuration file (e.g. cluster.yaml). In the image paths below, make sure to set <VERSION> to your cluster's version.
1
# cluster.yaml
2
​
3
image_manager: quay.io/cortexlabs/manager:<VERSION>
4
image_operator: quay.io/cortexlabs/operator:<VERSION>
5
image_downloader: quay.io/cortexlabs/downloader:<VERSION>
6
image_request_monitor: quay.io/cortexlabs/request-monitor:<VERSION>
7
image_cluster_autoscaler: quay.io/cortexlabs/cluster-autoscaler:<VERSION>
8
image_metrics_server: quay.io/cortexlabs/metrics-server:<VERSION>
9
image_nvidia: quay.io/cortexlabs/nvidia:<VERSION>
10
image_inferentia: quay.io/cortexlabs/inferentia:<VERSION>
11
image_neuron_rtd: quay.io/cortexlabs/neuron-rtd:<VERSION>
12
image_fluentd: quay.io/cortexlabs/fluentd:<VERSION>
13
image_statsd: quay.io/cortexlabs/statsd:<VERSION>
14
image_istio_proxy: quay.io/cortexlabs/istio-proxy:<VERSION>
15
image_istio_pilot: quay.io/cortexlabs/istio-pilot:<VERSION>
Copied!
For cluster version <= 0.20.0, also add the following two images:
1
image_istio_galley: quay.io/cortexlabs/istio-galley:<VERSION>
2
image_istio_citadel: quay.io/cortexlabs/istio-citadel:<VERSION>
Copied!
For Cortex cluster version < 0.16.0, please upgrade your cluster to the latest version.
Once you've updated your cluster configuration file, you can spin up your cluster (e.g. cortex cluster up --config cluster.yaml).
Update your API configuration file(s)
To configure your APIs to use the Quay images, you cna update your API configuration files. The image paths are specified in predictor.image (and predictor.tensorflow_serving_image for APIs with kind: tensorflow). Be advised that by default, the Docker Hub images are used for your predictors, so you will need to specify the Quay image paths for all of your APIs.
Here is a list of available images (make sure to set <VERSION> to your cluster's version):
1
quay.io/cortexlabs/python-predictor-cpu:<VERSION>
2
quay.io/cortexlabs/python-predictor-gpu:<VERSION>
3
quay.io/cortexlabs/python-predictor-inf:<VERSION>
4
quay.io/cortexlabs/tensorflow-serving-cpu:<VERSION>
5
quay.io/cortexlabs/tensorflow-serving-gpu:<VERSION>
6
quay.io/cortexlabs/tensorflow-serving-inf:<VERSION>
7
quay.io/cortexlabs/tensorflow-predictor:<VERSION>
8
quay.io/cortexlabs/onnx-predictor-cpu:<VERSION>
9
quay.io/cortexlabs/onnx-predictor-gpu:<VERSION>
10
quay.io/cortexlabs/python-predictor-cpu-slim:<VERSION>
11
quay.io/cortexlabs/python-predictor-gpu-slim:<VERSION>-cuda10.0
12
quay.io/cortexlabs/python-predictor-gpu-slim:<VERSION>-cuda10.1
13
quay.io/cortexlabs/python-predictor-gpu-slim:<VERSION>-cuda10.2
14
quay.io/cortexlabs/python-predictor-gpu-slim:<VERSION>-cuda11.0
15
quay.io/cortexlabs/python-predictor-inf-slim:<VERSION>
16
quay.io/cortexlabs/tensorflow-predictor-slim:<VERSION>
17
quay.io/cortexlabs/onnx-predictor-cpu-slim:<VERSION>
18
quay.io/cortexlabs/onnx-predictor-gpu-slim:<VERSION>
Copied!
Paid Docker Hub subscription
Another option is to pay for the Docker Hub subscription to remove the limit on the number of image pulls. Docker Hub's updated pricing model allows unlimited pulls on a Pro subscription for individuals as described here.
The advantage of this approach is that there's no need to do a cortex cluster down/cortex cluster up to authenticate with your Docker Hub account.
By default, the Cortex cluster pulls the images as an anonymous user. To configure your Cortex cluster to pull the images as an authenticated user, follow these steps:
Step 1
Install and configure kubectl (instructions).
Step 2
Set the following environment variables, replacing the placeholders with your docker username and password:
1
DOCKER_USERNAME=***
2
DOCKER_PASSWORD=***
Copied!
Run the following commands:
1
kubectl create secret docker-registry registry-credentials \
2
  --namespace default \
3
  --docker-username=$DOCKER_USERNAME \
4
  --docker-password=$DOCKER_PASSWORD
5
​
6
kubectl create secret docker-registry registry-credentials \
7
  --namespace kube-system \
8
  --docker-username=$DOCKER_USERNAME \
9
  --docker-password=$DOCKER_PASSWORD
10
​
11
kubectl patch serviceaccount default --namespace default \
12
  -p "{\"imagePullSecrets\": [{\"name\": \"registry-credentials\"}]}"
13
​
14
kubectl patch serviceaccount operator --namespace default \
15
  -p "{\"imagePullSecrets\": [{\"name\": \"registry-credentials\"}]}"
16
​
17
kubectl patch serviceaccount fluentd --namespace default \
18
  -p "{\"imagePullSecrets\": [{\"name\": \"registry-credentials\"}]}"
19
​
20
kubectl patch serviceaccount default --namespace kube-system \
21
  -p "{\"imagePullSecrets\": [{\"name\": \"registry-credentials\"}]}"
22
​
23
kubectl patch serviceaccount cluster-autoscaler --namespace kube-system \
24
  -p "{\"imagePullSecrets\": [{\"name\": \"registry-credentials\"}]}"
25
​
26
kubectl patch serviceaccount metrics-server --namespace kube-system \
27
  -p "{\"imagePullSecrets\": [{\"name\": \"registry-credentials\"}]}"
28
​
29
# Only if you are using cortex version <= 0.20.0:
30
kubectl patch serviceaccount istio-cni --namespace kube-system \
31
  -p "{\"imagePullSecrets\": [{\"name\": \"registry-credentials\"}]}"
32
​
33
# Only if you are using Inferentia:
34
kubectl patch serviceaccount neuron-device-plugin --namespace kube-system \
35
  -p "{\"imagePullSecrets\": [{\"name\": \"registry-credentials\"}]}"
Copied!
Updating your credentials
First remove your old docker credentials from the cluster:
1
kubectl delete secret --namespace default registry-credentials
2
kubectl delete secret --namespace kube-system registry-credentials
Copied!
Then repeat step 2 above with your updated credentials.
Removing your credentials
To remove your docker credentials from the cluster, run the following commands:
1
kubectl delete secret --namespace default registry-credentials
2
kubectl delete secret --namespace kube-system registry-credentials
3
​
4
kubectl patch serviceaccount default --namespace default -p "{\"imagePullSecrets\": []}"
5
kubectl patch serviceaccount operator --namespace default -p "{\"imagePullSecrets\": []}"
6
kubectl patch serviceaccount fluentd --namespace default -p "{\"imagePullSecrets\": []}"
7
kubectl patch serviceaccount default --namespace kube-system -p "{\"imagePullSecrets\": []}"
8
kubectl patch serviceaccount cluster-autoscaler --namespace kube-system -p "{\"imagePullSecrets\": []}"
9
kubectl patch serviceaccount metrics-server --namespace kube-system -p "{\"imagePullSecrets\": []}"
10
# Only if you are using cortex version <= 0.20.0:
11
kubectl patch serviceaccount istio-cni --namespace kube-system -p "{\"imagePullSecrets\": []}"
12
# Only if you are using Inferentia:
13
kubectl patch serviceaccount neuron-device-plugin --namespace kube-system -p "{\"imagePullSecrets\": []}"
Copied!
Push to AWS ECR (Elastic Container Registry)
You can also push the Cortex images to ECR on your AWS account, and pull from your ECR repository in your cluster. Follow this guide to do this.

Guides - Previous

Self-hosted Docker images

Next - Guides

Private docker registry

Last modified 10mo ago

Copy link

Contents

Use Cortex images from quay.io

Update your cluster configuration file

Update your API configuration file(s)

Paid Docker Hub subscription

Step 1

Step 2

Updating your credentials

Removing your credentials

Push to AWS ECR (Elastic Container Registry)