Inference Parsing
Making inference 🥶requires automated pipelines
When using nbox, you as a user do not need to worry about the kind of input you can send in. If it is an image model, you can pass it a URL and nbox would automatically manage the required pre-processing steps. All neural networks are matrix multiplications i.e. they consume numbers. However a URL is a string object, then how does nbox manage this operation. This document will give you more information on how we have built these operations without any hardcoding.
nbox comes preinstalled on our platform. For desktops pip install nbox would install all the required dependencies, however you would still need to install pytorch for running inference on your machine.

Primitives and Structures

Each input has two attributes to it, user does not define these but we internally do, a primitive and a structure:
    Primitives: This is the actual data you want your model to process. It can be of the following types:
      1.
      String like: These are filepaths,URLs and b64(only for images). This is how we think most people should use nbox. Why should the developer focus on conversion to and from tensors.
      2.
      Array like: This includes np.ndarray,list (for convenience called p-list). They are the mathematical values that are to be sent to your underlying model. Note: if your input is a tensor please convert that to numpy as tensor.numpy() for proper processing. This might seem unnecessary but strict input/output type checking provides guarantees to codebase this versatile.
      3.
      Binary like: This is slightly more complicated but this can be multipart/form-data(in case of upload for cloud-infer) or PIL.Imageobjects. More on this is given in the code.
    Structures: This is how data is organised, it is of three types:
      1.
      None: This is when the primitive is sent to the model directly
      2.
      Lists: In order to remove confusion this is called List. You can send:
        List[Primitives] like
      1
      [
      2
      "some/filepath",
      3
      "http://nice.image/url"
      4
      np.ones((400, 400, 3)),
      5
      PIL.Image.open("cat.jpg")
      6
      ]
      Copied!
        List[Dict] like
      1
      [
      2
      {"input_sentence": "Hello world"},
      3
      {"input_sentence": "Nice Chennai"}
      4
      ]
      Copied!
      3.
      Dictionaries: Your model might take in more than one input ex. bert and this requires feeding input as a dictionary. You can send:
        Dict[str, Primitives] like
      1
      {
      2
      "style_image": "http://nice.image/url",
      3
      "input_image": PIL.Image.open("cat.jpg")
      4
      }
      Copied!
        Dict[str, List[Primitives]] like
      1
      {
      2
      "input_sentence": "My name is BoJack"
      3
      "search_sentences": [
      4
      "A humanoid horse, BoJack Horseman",
      5
      "Wubba Lubba Dub Dub"
      6
      ]
      7
      }
      Copied!
For any kind of input the parser generates an intermediate representation in numpy which is why it is given as a dependency.

Strict Checking

Though the input to python code is completely unchecked, we do have strict internal checking. This list includes, but is not limited to:
    1.
    Batching: When input is more than one, the output arrays are concatenated to create a batch. This will throw an error if there is a shape mismatch.
    2.
    Key matching: When we get list of dict, all the keys should match.
    3.
    Numpy Tensors: Though the underlying model can be from any framework, it is important to keep the input/output properly type checked. If your input is an array please convert that to a numpy array.

Parsers

nbox comes with a few parsers packaged right now for images and text.

nbox.parsers.ImageParser()

Single unified Image parser that consumes different types of data and returns a processed numpy array.
Args:
    post_proc_fn: function that takes in the numpy array and returns some processed values. By default we use lambda x: torch.from_numpy(x) for conversion to torch.Tensor that is then passed for local inference
    cloud_infer: If True this parser normalises values between [-1, 1]

nbox.parsers.TextParser()

Unified Text parsing engine, returns tokenized dictionaries. Currently this is built with hf.co/transformers models in mind.
Args:
    tokenizer: transformers.AutoTokenizer object or any __call__-able method or class that returns a dictionary with lists
    post_proc_fn: function that takes in the numpy array and returns some processed values. By default we use lambda x: torch.from_numpy(x) for conversion to torch.Tensor that is then passed for local inference

Next Steps

How to add your own Parser?

Adding your own parser is very simple. You have to inherit class nbox.parsers.BaseParser and define the following functions (for eg. consider stock price parser):
    process_list(): How to process input object that is of type list. In case of stock prices the input can be actual list of closing values in which case the input is a p-list or the input can be list of dictionaries.
    process_dict(): How to process input object that is of type dictionaries. In case of stock prices this can be dictionaries with different indices.
    process_primitive(): How to process any individual primitive for your input.
The BaseParser automatically manages the required calls based on the inputs type. You can also perform custom post_
Last modified 24d ago