Skip to main content

Use fastapi for model inference over http

Project description

model-inference-fastapi

Running inference on ML model over http using fastapi

Installation

pip install model-inference-fastapi

Usage

  1. Create a model service file model_service.py implementing the ModelServiceBase interface like so.

    from typing import Dict, List, Optional
    from model_inference_fastapi.model_service_base import ModelServiceBase
    
    
    class ModelService(ModelServiceBase):
        __model = None
    
        @staticmethod
        def validate(data: Optional[List[Dict]] = None, image_files: Optional[List[bytes]] = None) -> Optional[str]:
            return
    
        @staticmethod
        def predict(data: Optional[List[Dict]] = None, image_files: Optional[List[bytes]] = None) -> List[Dict]:
            return [{"status": "Ready to Go!!!"}]
    

    For instance if you are trying to deploy an image classification model ( e.g. ResNet50) using torch. You will need the following model_service.py example in pytorch.

    import json
    from typing import Dict, List, Optional
    
    import torch
    from torchvision.models import resnet50
    from torchvision.transforms import Resize, Compose, CenterCrop, ToTensor, Normalize
    
    from model_inference_fastapi.model_service_base import ModelServiceBase
    from model_inference_fastapi.utils import convert_file_to_pil_image
    
    
    # derived from https://keras.io/api/applications/
    class ModelService(ModelServiceBase):
        __model = resnet50(weights="IMAGENET1K_V2")
        __transform = Compose([
            Resize(size=256),
            CenterCrop(size=224),
            ToTensor(),
            Normalize([0.485, 0.456, 0.406],
                      [0.229, 0.224, 0.225])
        ])
    
        with open("imagenet1000_clsidx_to_labels.txt") as f:
            __class_names = eval(f.read())
    
        @staticmethod
        def validate(*, data: Optional[List[Dict]] = None, image_files: Optional[List[bytes]] = None):
            if len(image_files) != 1:
                return "There must be at least one image file in image_files"
    
        @staticmethod
        def predict(*, data: Optional[List[Dict]] = None, image_files: Optional[List[bytes]] = None) -> List[Dict]:
            test_image_tensor = ModelService.__transform(convert_file_to_pil_image(image_files[0]))
            test_image_tensor = test_image_tensor.view(1, 3, 224, 224)
    
            with torch.no_grad():
                ModelService.__model.eval()
                # Model outputs log probabilities
                out = ModelService.__model(test_image_tensor)
                ps = torch.exp(out)
                top_k, top_class = ps.topk(1, dim=1)
                print("Output class :  ", ModelService.__class_names[top_class.cpu().numpy()[0][0]])
    
            return [{"class": ModelService.__class_names[top_class.cpu().numpy()[0][0]]}]
    
  2. Run Server

    uvicorn model_inference_fastapi.main:app
    
  3. Visit the API testing page powered by FastAPI here http://127.0.0.1:8000/docs/.

    • Navigate to the /predict/ AKA Prediction Route dropdown
    • Click the Try it out button
    • For the image_files array click Add string item button
    • Choose a file to upload
    • Click execute
    • Scroll down to see the server response

Testing model_service.py

Here is a sample test file for image classification model_service.py

# replace this import with import for you own `model_service.py`
from model_inference_fastapi.model_service import ModelService

def test_predict():
    with open('/path/to/image.jpg', 'rb') as file:
        prediction = ModelService.predict(image_files=[file.read()])
        assert prediction[0]["class"] == "dog" 

Using the docker image

See example dir for more info.

  1. create a Dockerfile like so
FROM robotstech/model-inference-fastapi

EXPOSE 8000
WORKDIR /app

COPY requirements.txt ./requirements.txt

RUN pip install --upgrade pip
RUN pip install -r requirements.txt

# move model_service.py
COPY ./model_service.py ./model_inference_fastapi/

# move data or other resources to root dir of model-inference
COPY ./imagenet1000_clsidx_to_labels.txt .

The key info here is to move the model_service.py and data/resource to the right directory

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

model_inference_fastapi-0.0.1a0.tar.gz (63.4 kB view hashes)

Uploaded Source

Built Distribution

model_inference_fastapi-0.0.1a0-py3-none-any.whl (7.2 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page