Error in Invoking Universal Sentence Encoder Sagemaker Endpoint

[singularity014]

Describe the bug
Hello,
I deployed a multilingual universal sentence encoder model on using SageMaker notebook instance.
The model got deployed successfully. But when I am trying to predict it from the notebook instance.
I am getting the following error -

ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (404) from model with message "{ "error": "[Derived]{{function_node __inference_signature_wrapper_227768}} {{function_node __inference_signature_wrapper_227768}} Op type not registered 'SentencepieceOp' in binary running on model.aws.local. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) tf.contrib.resampler should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.\n\t [[{{node StatefulPartitionedCall}}]]\n\t [[StatefulPartitionedCall]]" }". See https://eu-west-1.console.aws.amazon.com/cloudwatch/home?region=eu-west-1#logEventViewer:group=/aws/sagemaker/Endpoints/museA in account 647725471721 for more information.

To reproduce
from sagemaker import get_execution_role
import sagemaker
role = get_execution_role()
from sagemaker.tensorflow.serving import Model, Predictor

sagemaker_model = Model(
model_data = 's3://naister-platform-models/universal-sentence-encoder-multilingual-large_3.tar.gz',
role = role,
framework_version='2.0.0'
)

predictor = sagemaker_model.deploy(
initial_instance_count=1,
instance_type='ml.t2.medium',
endpoint_name="museA"
)

input = ['Winter is coming', 'Hold the door, Hodor']
output = predictor.predict(input)

Expected behavior
Sentence Embeddings

System information
A description of your system. Please provide:

• Using Sagemaker Notebook type conda_amazonei_tensorflow2_p36:
• Framework name ( Tensorflow), model (Multilingual Universal sentence encoder large):
• Framework version: 2.2.0
• Python version: 3.6

Assistance requested regarding this problem.

[chuyang-deng]

Hi @singularity014,

The ModelError you had seems to be a TensorFlow specific error, which I've found a potential solution to it: tensorflow/hub#463

Are you using an inference.py script for predictions? If so you could try adding tensorflow_text and import it in the inference script.

Here's instruction on how to bring in custom modules (towards the end of the Pre/Post-Processing section):
https://github.com/aws/sagemaker-tensorflow-serving-container#prepost-processing

[singularity014]

Hi..I think it is because of the sentence piece and tensorflow-text missing dependency from the Tensorflow Hub container used by SageMaker.