Convert Models to OpenVINO Format

This page explains how to convert various generative AI models from Hugging Face and ModelScope to OpenVINO IR format. Refer to the Supported Models for a list of available models.

For downloading pre-converted models, see Download Pre-Converted OpenVINO Models.

Converting Models from Hugging Face

1. Install optimum-intel package to download, convert and optimize models:

   pip install optimum-intel@git+https://github.com/huggingface/optimum-intel.git

2. Download and convert a model to the OpenVINO IR format using optimum-cli tool from Hugging Face:

   optimum-cli export --model <model_id_or_path> <output_dir>
   tip

   For better performance with minimal accuracy impact, convert the model to lower precision by using --weight-format argument:

   INT4

   optimum-cli export --model <model_id_or_path> --weight-format int4 <output_dir>

   INT8

   optimum-cli export --model <model_id_or_path> --weight-format int8 <output_dir>

   FP16

   optimum-cli export --model <model_id_or_path> --weight-format fp16 <output_dir>
   info

   The --trust-remote-code flag is required for some models that use custom code.

   Check a full list of conversion options here.

Converting Models from ModelScope

ModelScope models need to be downloaded first, then converted to OpenVINO IR format.

1. Install modelscope and optimum-intel packages to download, convert and optimize models:

   pip install modelscope
   pip install optimum-intel@git+https://github.com/huggingface/optimum-intel.git
2. Download the required model (e.g. Qwen/Qwen2-7b) to a local directory using modelscope tool:

   modelscope download --model 'Qwen/Qwen2-7b' --local_dir <model_path>
3. Convert the model (and optionally compress weights) using optimum-cli tool:

   optimum-cli export --model <model_path> --weight-format int4 <output_dir>