Example

HTTP
Create HTTP APIs that respond to requests in real-time.
Implement
mkdir text-generator && cd text-generator
touch handler.py requirements.txt text_generator.yaml
# handler.py
​
from transformers import pipeline
​
class Handler:
    def __init__(self, config):
        self.model = pipeline(task="text-generation")
​
    def handle_post(self, payload):
        return self.model(payload["text"])[0]
# requirements.txt
​
transformers
torch
# text_generator.yaml
​
- name: text-generator
  kind: RealtimeAPI
  handler:
    type: python
    path: handler.py
  compute:
    gpu: 1
Deploy
cortex deploy text_generator.yaml
Monitor
cortex get text-generator --watch
Stream logs
cortex logs text-generator
Make a request
curl http://***.elb.us-west-2.amazonaws.com/text-generator -X POST -H "Content-Type: application/json" -d '{"text": "hello world"}'
Delete
cortex delete text-generator
gRPC
To make the above API use gRPC as its protocol, make the following changes (the rest of the steps are the same):
Add protobuf file
Create a handler.proto file in your project's directory:
<!-- handler.proto -->
​
syntax = "proto3";
package text_generator;
​
service Handler {
    rpc Predict (Message) returns (Message);
}
​
message Message {
    string text = 1;
}
Set the handler.protobuf_path field in the API spec to point to the handler.proto file:
# text_generator.yaml
​
- name: text-generator
  kind: RealtimeAPI
  handler:
    type: python
    path: handler.py
    protobuf_path: handler.proto
  compute:
    gpu: 1
Match RPC service name
Match the name of the RPC service(s) from the protobuf definition (in this case Predict) with what you're defining in the handler's implementation:
# handler.py
​
from transformers import pipeline
​
class Handler:
    def __init__(self, config, proto_module_pb2):
        self.model = pipeline(task="text-generation")
        self.proto_module_pb2 = proto_module_pb2
​
    def Predict(self, payload):
        return self.proto_module_pb2.Message(text="returned message")
Make a gRPC request
grpcurl -plaintext -proto handler.proto -d '{"text": "hello-world"}' ***.elb.us-west-2.amazonaws.com:80 text_generator.Handler/Predict
Workloads - Previous
Realtime APIs
Handler
Last updated 1 week ago