Evaluation Methodology Vivo Ai Lab Smartbench Deepwiki

Evaluation Methodology Vivo Ai Lab Smartbench Deepwiki This page documents smartbench's evaluation methodology, which uses an llm as a judge approach to automatically assess model performance across 20 tasks. the methodology includes evaluation prompt design, scoring dimensions, validation procedures, and multi precision testing protocols. We conduct comprehensive evaluations of on device llms and mllms using smartbench and also assess their performance after quantized deployment on real smartphone npus.

Vivo Ai Lab Iqoo To tackle these gaps, in this paper, starting from a functional investigation of on device llms, we construct smartbench, the first (chinese) benchmark for evaluating the capabilities of on device llms in mobile scenarios. This page introduces smartbench, a benchmark system for evaluating chinese smartphone device side large language models (llms). it explains the benchmark's purpose, key features, system components, and evaluation methodology. This page provides an overview of smartbench's three stage evaluation pipeline, which is the core execution workflow for assessing device side llms. the pipeline transforms raw test data into final performance scores through three sequential python scripts: generate results.py, evaluate results.py, and process results.py. For information on the underlying evaluation pipeline architecture, see evaluation pipeline. scope: this guide focuses on the practical aspects of running evaluations.

Using Custom Models Vivo Ai Lab Smartbench Deepwiki This page provides an overview of smartbench's three stage evaluation pipeline, which is the core execution workflow for assessing device side llms. the pipeline transforms raw test data into final performance scores through three sequential python scripts: generate results.py, evaluate results.py, and process results.py. For information on the underlying evaluation pipeline architecture, see evaluation pipeline. scope: this guide focuses on the practical aspects of running evaluations. This guide walks you through the initial setup and execution of your first evaluation with smartbench. you will learn how to install dependencies, configure the directory structure, and run the three stage evaluation pipeline to assess a device side llm model. This section covers advanced usage scenarios and extensibility features of smartbench beyond the basic three stage evaluation workflow. it addresses customization, integration of new components, multi precision evaluation setups, and error handling strategies for production deployments and research experimentation. This document describes the high level architecture of smartbench, including its three stage evaluation pipeline, data flow between components, file formats, and external dependencies. Smartbench 是第一个专门针对中文智能手机场景下设备端大语言模型（llm）能力评估的基准。 它通过分析智能手机制造商提供的功能，将设备端 llm 功能分为五个类别，共 20 个具体任务，涵盖了文本摘要、文本问答、信息抽取、内容创作和通知管理等实际应用场景。.

Github Vivo Ai Lab Smartbench Emnlp 2025 Smartbench Is Your Llm This guide walks you through the initial setup and execution of your first evaluation with smartbench. you will learn how to install dependencies, configure the directory structure, and run the three stage evaluation pipeline to assess a device side llm model. This section covers advanced usage scenarios and extensibility features of smartbench beyond the basic three stage evaluation workflow. it addresses customization, integration of new components, multi precision evaluation setups, and error handling strategies for production deployments and research experimentation. This document describes the high level architecture of smartbench, including its three stage evaluation pipeline, data flow between components, file formats, and external dependencies. Smartbench 是第一个专门针对中文智能手机场景下设备端大语言模型（llm）能力评估的基准。 它通过分析智能手机制造商提供的功能，将设备端 llm 功能分为五个类别，共 20 个具体任务，涵盖了文本摘要、文本问答、信息抽取、内容创作和通知管理等实际应用场景。.