ML demo development made easy with HuggingFace
I’ve heard a lot of good words about HuggingFace on how user-friendly their products are. I’ve been very itching to try it myself for a while and finally got my chance to work on it this weekend.
I planned to test model checkpoint serving using HF hub and demo deployment using HF space.
The overall experience was very encouraging. I managed to get something working locally within 1 hour and spend a few more hours solving the dependency issues for deployment. I’ll cover that later.
Here I just take the OCR model from MMLab as an exercise. The MMLab team open sourced a variety of state-of-art Computer Vision models in OpenMMLab repo, including Classification, Detection, Segmentation, etc., and provided an easy-to-use API for each task.
Model checkpoints serving
We want to store and serve some model weights on HuggingFace hub. The HF hub is an ML practitioner’s Github with optimizations on large files performance such as model checkpoints.
To do that, we need to first find where the checkpoints are stored in MMOCR.
Running the MMOCR_Tutorial.ipynb demo gives some hints on the current checkpoints download path.
load checkpoint from http path: https://download.openmmlab.com/mmocr/textdet/panet/panet_r18_fpem_ffm_sbn_600e_icdar2015_20210219-42dbe46a.pth
load checkpoint from http path: https://download.openmmlab.com/mmocr/textrecog/seg/seg_r31_1by16_fpnocr_academic-72235b11.pthA quick search on the“.pth”s suggests a few potential locations where these paths were defined. A few trial-and-error can help us locating the actual definition in ocr.py
I made a simple change to load the default weights from a Hugging Face model repo instead.
This is a great video explaining how to create a repo on HuggingFace and upload files to it. There are two videos on this topic on the official Youtube HuggingFace account. And the link above is the most recent one. The process is pretty straightforward. Just to note that the code snippet in 4:02 is not working. The API has been changed recently and all arguments are now named arguments.
Now run the demo again, and we could see that the checkpoints are downloaded from HF.
load checkpoint from http path: https://huggingface.co/xianbao/mmocr/resolve/main/panet_r18_fpem_ffm_sbn_600e_icdar2015_20210219-42dbe46a.pth
load checkpoint from http path: https://huggingface.co/xianbao/mmocr/resolve/main/seg_r31_1by16_fpnocr_academic-72235b11.pthThe final code can be found from my fork in https://github.com/xianbaoqian/mmocr
Gradio demo
Gradio is an innovative, super simple framework to convert ML code to a nice UI demo. And let’s see how to use this library to create an OCR demo.
The command line demo code is pretty simple, thanks to MMOCR’s high level interface.
from mmocr.utils.ocr import MMOCR# Load models into memory
ocr = MMOCR()# Inference
results = ocr.readtext('mmocr/demo/demo_text_ocr.jpg', print_result=True, imshow=True)
And all we need to do is to define a list of interfaces (inputs, outputs) and wrap ocr.readtextcall in a function that gets triggered every time user submits an image.
Here is the final code. Note that MMOCR expects a config folder, which has been excluded from the MMOCR pypi package and needs to be copied from the repo.
import mmocr
import gradio as gr
import os
from mmocr.utils.ocr import MMOCRconfig_dir = os.path.join(os.path.dirname(__file__), 'configs/')# TODO: Put more models on HF hub.
ocr = MMOCR(config_dir=config_dir)def infer(image):
# TODO: Also display bounding boxes interactively
return ocr.readtext(image, output='.', print_result=True, imshow=False)# TODO: a drop down list for model selection
iface = gr.Interface(fn=infer, inputs="image", outputs="json")iface.launch()
The final app.py file can be found from https://huggingface.co/spaces/xianbao/mmocr-demo/blob/main/app.py
Demo deployment on HF spaces
Everything went super smooth so far, but this deployment step actually took me quite some time trying to figure out how to solve the MMOCR dependency issues on HF spaces.
HF spaces allows both Python dependencies and Debian dependencies. Upon each git upload the repo will be built as a Docker container to serve traffic. Here is an example of BUILD log and serving log.
My first attempt was to just put everything that I need in the requirements.txt But that doesn’t work.
Traceback (most recent call last):
File "app.py", line 5, in <module>
from mmocr.utils.ocr import MMOCR
File "/home/user/.local/lib/python3.8/site-packages/mmocr/utils/__init__.py", line 2, in <module>
from mmcv.utils import Registry, build_from_cfg
ImportError: cannot import name 'Registry' from 'mmcv.utils' (/home/user/.local/lib/python3.8/site-packages/mmcv/utils/__init__.py)Surprise. Surprise. Everything works fine locally. There must be something wrong with how MMCV dependency was installed. Luckily HF provided the entire docker container build log, so I was able to try replicating the same build environment locally.
After reading MMOCR’s doc and playing a few alternative solutions, I realized that openmim is probably the best tool to install mm* dependencies. I came up with this Dockerfile that works perfectly locally so I uploaded it to HF spaces and hoping that HF spaces could pick up my customized Dockerfile instead of the default one.
Unfortunately, HF spaces doesn’t seem to like customized Dockerfile so this approach doesn’t work, at least for my case.
How am I supposed to run a shell command prior to launching the server?
I came across this example that calls os.system to run a command upon server start. I believe that this might work, but not ideal, because the dependency installation happened at runtime during service startup, not at build time. As the MMCV installation is slow, this approach could cause a huge delay on cold service startup, up to 10 minutes. Although there is no documentation explaining the life time management of HF spaces container, I would assume that service startup would be ran more than once as it’s a waste of resources to keep all the containers running and waiting for incoming requests.
I would prefer to finish the entire installation process only once when building the docker image.
Most of the PyPI installation failures came with version mismatch. Could that be the case here as well? After pinning the version in requirements.txt, I now get a different error:
/usr/local/lib/python3.8/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: libc10_hip.so: cannot open shared object file: No such file or directory
warn(f"Failed to load image Python extension: {e}")
Traceback (most recent call last):
File "/home/user/app/app.py", line 5, in <module>
from mmocr.utils.ocr import MMOCR
File "/usr/local/lib/python3.8/site-packages/mmocr/utils/ocr.py", line 24, in <module>
from mmocr.apis import init_detector
File "/usr/local/lib/python3.8/site-packages/mmocr/apis/__init__.py", line 2, in <module>
from .inference import init_detector, model_inference
File "/usr/local/lib/python3.8/site-packages/mmocr/apis/inference.py", line 7, in <module>
from mmcv.ops import RoIPool
File "/usr/local/lib/python3.8/site-packages/mmcv/ops/__init__.py", line 2, in <module>
from .active_rotated_filter import active_rotated_filter
File "/usr/local/lib/python3.8/site-packages/mmcv/ops/active_rotated_filter.py", line 10, in <module>
ext_module = ext_loader.load_ext(
File "/usr/local/lib/python3.8/site-packages/mmcv/utils/ext_loader.py", line 13, in load_ext
ext = importlib.import_module('mmcv.' + name)
File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ImportError: /usr/local/lib/python3.8/site-packages/mmcv/_ext.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZNK2at6Tensor6narrowElllUndefined symbol! That’s something I know! It happens when the Python library is calling a native method in a dynamic library, e.g. .so file, with a mismatched version. The mangled C++ method name seems to be Tensor::Narrow from Pytorch, so could that be a version mismatch between MMCV and PyTorch?
It is indeed the case, and here is the requirements.txt that does the magic.
--find-links https://download.pytorch.org/whl/torch_stable.html
torch==1.12.0+cpu
torchvision--find-links https://download.openmmlab.com/mmcv/dist/cpu/torch1.12.0/index.html
mmcv-full==1.6.0
mmdet==2.25.0
And finally you can find the final app here: https://huggingface.co/spaces/xianbao/mmocr-demo
This is amazing, isn’t it? You don’t need to write a single line of Javascript or any backend service that you have to maintain, yet you get a full-featured UI demo running on the internet. And everything is free of charge. ❤️
Hope you find this article useful and feel free to leave comments/feedbacks.