https://llm.mlc.ai/docs/install/mlc_llm
环境安装
conda create -n mlc_llm python=3.10
Mac cpu+metal
macuser03@macuser03noMacBook-Pro study_001 % conda activate mlc_llm
(mlc_llm) macuser03@macuser03noMacBook-Pro study_001 % python -m pip install --pre -U -f https://mlc.ai/wheels mlc-llm-nightly mlc-ai-nightly
Always check if conda is installed properly in macOS using the command below:
(mlc_llm) macuser03@macuser03noMacBook-Pro study_001 % conda info | grep platform
platform : osx-64
It should return “osx-64” for Mac with Intel chip, and “osx-arm64” for Mac with Apple chip. We need git-lfs in the system, you can install it via
conda install -c conda-forge git-lfs
Then you can verify installation in command line:
python -c "import mlc_llm; print(mlc_llm)"
# Prints out: <module 'mlc_llm' from '/path-to-env/lib/python3.11/site-packages/mlc_llm/__init__.py'>
Chat CLI
(mlc_llm) macuser03@macuser03noMacBook-Pro study_001 % python -c "import mlc_llm; print(mlc_llm)"
<module 'mlc_llm' from '/usr/local/Caskroom/miniconda/base/envs/mlc_llm/lib/python3.10/site-packages/mlc_llm/__init__.py'>
(mlc_llm) macuser03@macuser03noMacBook-Pro study_001 % mlc_llm chat HF://mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC
[2024-08-20 13:36:52] INFO auto_device.py:88: Not found device: cuda:0
[2024-08-20 13:36:53] INFO auto_device.py:88: Not found device: rocm:0
[2024-08-20 13:36:55] INFO auto_device.py:79: Found device: metal:0
[2024-08-20 13:36:55] INFO auto_device.py:79: Found device: metal:1
[2024-08-20 13:36:57] INFO auto_device.py:88: Not found device: vulkan:0
[2024-08-20 13:36:58] INFO auto_device.py:88: Not found device: opencl:0
[2024-08-20 13:36:58] INFO auto_device.py:35: Using device: metal:0
[2024-08-20 13:36:58] INFO download_cache.py:227: Downloading model from HuggingFace: HF://mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC
[2024-08-20 13:36:58] INFO download_cache.py:29: MLC_DOWNLOAD_CACHE_POLICY = ON. Can be one of: ON, OFF, REDO, READONLY
[2024-08-20 13:36:58] INFO download_cache.py:56: [Git] Cloning https://huggingface.co/mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC.git to /var/folders/36/jzmw3wbd47g3601swp3362500000gn/T/tmpr1lvqwsz/tmp
[2024-08-20 13:37:01] INFO download_cache.py:92: [Git LFS] Downloading 0 files with Git LFS: []