Using GPUs

To use GPUs:
1.Make sure your AWS account is subscribed to the EKS-optimized AMI with GPU Support.
2.You may need to file an AWS support ticket to increase the limit for your desired instance type.
3.Set instance type to an AWS GPU instance (e.g. g4dn.xlarge) when installing Cortex.
4.Set the gpu field in the compute configuration for your API. One unit of GPU corresponds to one virtual GPU. Fractional requests are not allowed.
Tips
If using processes_per_replica > 1, TensorFlow-based models, and Python Predictor
When using processes_per_replica > 1 with TensorFlow-based models (including Keras) in the Python Predictor, loading the model in separate processes at the same time will throw a CUDA_ERROR_OUT_OF_MEMORY: out of memory error. This is because the first process that loads the model will allocate all of the GPU's memory and leave none to other processes. To prevent this from happening, the per-process GPU memory usage can be limited. There are two methods:
1) Configure the model to allocate only as much memory as it requires, via tf.config.experimental.set_memory_growth():
1
for gpu in tf.config.list_physical_devices("GPU"):
2
    tf.config.experimental.set_memory_growth(gpu, True)
Copied!
2) Impose a hard limit on how much memory the model can use, via tf.config.set_logical_device_configuration():
1
mem_limit_mb = 1024
2
for gpu in tf.config.list_physical_devices("GPU"):
3
    tf.config.set_logical_device_configuration(
4
        gpu, [tf.config.LogicalDeviceConfiguration(memory_limit=mem_limit_mb)]
5
    )
Copied!
See the TensorFlow GPU guide and this blog post for additional information.

Advanced - Previous

Compute

Next - Advanced

Using Inferentia

Last modified 10mo ago

Copy link

Contents

Tips

If using processes_per_replica > 1, TensorFlow-based models, and Python Predictor