NimbleBox.ai started to reduce entry barrier to production grade AI workloads for developers. We started that by introducing our platform, and now with our SDK
nbox. Inference of models is the most common operation we see on our platform, and is a bottleneck for non-ML developers who just want to use that as a function and don't want to setup a pipeline.
Find the complete open source code on Github. Install the package by running the command:
pip install nbox
nbox for the first time, it will prompt you the username and password and create a secrets file at
~/.nbx/secrets.json. This file then contains all the information that you don't have to fetch manually again.
nbox is a new package designed ground up with inference and production grade deployment in mind. The input to model is called
input_object and it can be a string, array-like, binary-like in form of a list or Dict. Read full technical document here.
import nbox# As all these models come from the popular frameworks you use such as# torchvision, efficient_pytorch or hf.transformersmodel = nbox.load("torchvision/mobilenetv2", pretrained = True)# nbox makes inference the priority so you canout = model('cat.jpg') # pass it image pathout = model(np.array(Image.open('cat.jpg'))) # pass it numpy arraysout = model(['cat.jpg', Image.open('cat.jpg')]) # pass it a list for batch inference
nbox can load any underlying model from package and can consume anything (eg. code above) whether it is
PIL.Image or just a
This package can always be used as a standalone package however it also integrates tightly with NimbleBox.ai's deployment services. This is how easy it is to deploy your model using
nbox. Read more about deployment in API documentation.
# load any model and simply provide your API key and machine_idmodel = nbox.load("transformers/gpt2", pretrained = True)url_endpoint, api_key = model.deploy(("This is some sample input string, it can be ""numpy array or list of strings. You don't need ""to take the headache of conversion to strings."))
Once deployment starts see your terminal light up!
Often your local machine does not have enough power to run large models that you need for your task. Instead use cloud infer for this and feel no difference in experience, but in the background you will make an API call to NimbleBox.ai managed deployments. When loading the model simply pass the URL and keys from your dashboard.
Copy the URL and API key from the dashboard and simply pass that to
[WIP] We have support for most common pre-post operations from inside
nbox however python is too slow for production services and so we are building Rust intermediate layers that will handle intermediate workloads using multicore code, giving very low latencies.
# load by passing the URL and nbx_api_keymodel = nbox.load("https://api.nimblebox.ai/user/my_big_model/","nbxdeploy_zpO8I8AVzvOetQYAZanzP2mMgJ5oh84LG0wZdgh3U")# use as if it runs locallypred = model("https://s3.aws.com/bucket/my_info.jpg")out = model('cat.jpg') # pass it image pathout = model(np.array(Image.open('cat.jpg'))) # pass it numpy arraysout = model(['cat.jpg', Image.open('cat.jpg')]) # pass it a list for batch inference
nbox has all the things required for image and text based models, you can add your own custom pipelines by writing just one function. Read more in documentation:
How to add your own parsers? link
Deploying your own model link
Loading your own model link
Raise Feature Requests link
Hope you enjoy this and it makes your life easier!