FastChat 竟然真的可以单机用,却也没法用

[paopjian]

https://github.com/lm-sys/FastChat 我单机用了 51g 内存跑起来了,就是速度慢得离谱,毕竟是 CPU,用英文对话可以一个字一个字得往外蹦,差不多一个词 10s,我这还是 8700,要是换个更好的可能速度更快.

中文就更慢了,不知道是卡了还是怎么回事,20 分钟不出一句话,huggingface 里的 demo 倒是可以中文.

这是跑了几分钟的结果

Human: write a vue demo
Assistant: Here is a simple demo of a Vue.js component that displays "Hello, World!" in the center of the screen:

<template>
 <div>
 <h1>Hello, World!</h1>
 </div>
</template>

<script>
 export default {
 mounted() {
 console.log('The component has been mounted.');
 }
 }
</script>

This component uses Vue's template syntax to define the HTML structure of the component, and the script section contains
 the Vue logic for the component. The `mounted` function is called when the component is mounted, and it logs a message to the console. When the component is used in a Vue application, the `Hello, World!` message will be displayed in the center of the screen.
(用时 10 分钟)

Human: "你好" translate to English
Assistant: "你好" is the Chinese word for "hello." In English, it is pronounced as "you hao."
(用时 2 分钟)

Human: <水浒传>是谁写的
Assistant: ravinder singh
Human: Who is that?
Assistant: Ravinder Singh is an Indian author. He is known for writing the novel "The Water's Edge," which tells the story of a young woman's journey to find love and fulfillment in the midst of challenging circumstances. The novel has been
 translated into multiple languages and has received critical acclaim.
 (用时 5 分钟)

后面结果全错了,小模型想比大模型确实在"知识量"上有差距,不过我对未来的 LLM 非常有信心,我觉得显卡厂可以出一张 AI 专门卡,超大显存就为加载 LLM,但是不带训练功能,只为个人用户本地运行大语言模型

PS:模型还需要先从 LLaMA 加权重转换,这时候内存占用超过 60g 了,差一点都不行,压缩倒是 51g 就可以了.

[superares]

你用 q4.0 的模型了吗？那个很小