1. 前言
非常幸运得到这次测评 CSK6 视觉AI开发套件的机会,在这里先感谢极术社区和聆思科技举办的活动。这次试用下来,从工具链、SDK、文档到示例,让我体会到了聆思科技在 AI 芯片深厚的技术积累,我只有一个评价:专业 👍。
2. CSK6芯片简介
CSK6 是聆思科技新一代的 AI 芯片 SoC 产品系列,采用多核异构架构,集成了 ARM Star MCU,HiFi4 DSP,以及聆思全新设计的 AI 神经网络处理内核 NPU,算力达到 128 GOPS。多核异构的设计使芯片能以较低功耗满足音频及图像视频的 AI 应用需求。
本系列芯片集成了 SRAM 与 PSRAM,支持内置或外接Flash,可提供最高 4 入 2 出的 Audio Codec,VGA 像素的 DVP 摄像头接口,多达 6 路的触控检测以及 SPI、UART、USB、SDIO、I2C、I2S 等各类外设接口,丰富接口支持各类应用方案的开发。
3. 开发环境搭建
CSK6 的开发环境搭建非常简单,只需要参考 官方文档
我个人使用家里工作站上的 ubuntu 20.04 进行编译,PC 的 windows 11 用来烧录和查看,代码和编译结果都在 NAS 上,所以这里遇到了一点问题:
官方提供的离线包是 snap 包,而 snap 不支持访问某些自行 mount 的目录,也就没法访问我的 NAS 😂。
于是我还是采用了在线安装的方式:
curl -o- https://cdn.iflyos.cn/public/cskTools/lisa-zephyr-install.sh | bash
SDK 会安装在 ~/.listenai
目录。
4. 人脸识别
clone sample
lisa zep create --from-git https://cloud.listenai.com/zephyr/applications/app_algo_fd_sample_for_csk6.git
编译固件
lisa zep build -b csk6011a_nano
看到如下显示就表示成功了
[248/248] Linking C executable zephyr/zephyr.elf Memory region Used Size Region Size %age Used FLASH: 191040 B 16 MB 1.14% SRAM: 127504 B 320 KB 38.91% ITCM: 6268 B 16 KB 38.26% DTCM: 64 B 16 KB 0.39% PSRAMAP: 1591536 B 2164 KB 71.82% PSRAMAP_NOCACHE: 0 GB 1804 KB 0.00% IDT_LIST: 0 GB 2 KB 0.00% ✔ 构建成功
烧录固件和资源
我需要在 windows 上查看结果,可以使用 cskburn,可以在 GitHub release 下载
串口查看工具可以使用免费的 MobaXterm。
烧录 DSP 固件和模型资源(base) PS D:\Software\Development\listenai> .\cskburn -s \\.\COM5 -b 748800 -C 6 0x400000 "X:\work\repo\csk6-learn\app_algo_fd_sample_for_csk6\resource\cp.bin" 0x500000 "X:\work\repo\csk6-learn\app_algo_fd_sample_for_csk6\resource\res.bin" Partition 1: 0x00400000 (783.35 KB) - X:\work\repo\csk6-learn\app_algo_fd_sample_for_csk6\resource\cp.bin Partition 2: 0x00500000 (7003.59 KB) - X:\work\repo\csk6-learn\app_algo_fd_sample_for_csk6\resource\res.bin Waiting for device... Entering update mode... Detected flash size: 16 MB Burning partition 1/2... (0x00400000, 783.35 KB) 783.35 KB / 783.35 KB (100.00%) Burning partition 2/2... (0x00500000, 7003.59 KB) 7003.59 KB / 7003.59 KB (100.00%) Finished
烧录固件
(base) PS D:\Software\Development\listenai> .\cskburn -s \\.\COM5 -b 748800 -C 6 0x0 "X:\work\repo\csk6-learn\app_algo_fd_sample_for_csk6\build\zephyr\zephyr.bin" Partition 1: 0x00000000 (186.56 KB) - X:\work\repo\csk6-learn\app_algo_fd_sample_for_csk6\build\zephyr\zephyr.bin Waiting for device... Entering update mode... Detected flash size: 16 MB Burning partition 1/1... (0x00000000, 186.56 KB) 186.56 KB / 186.56 KB (100.00%) Finished
PC 查看结果
拉取查看工具git clone https://cloud.listenai.com/zephyr/applications/csk_view_finder_spd.git
按照提示安装驱动选择设备并连接
就可以实时查看识别结果了
5. tflite-micro 语音识别推理测试
新建项目
lisa zep create tflite_micro --from hello_world
选择 hello-world,输入项目名 tflite_micro
启用 tflite-micro 和 C++ 支持
启动 menuconfiglisa zep build -t menuconfig -b csk6011a_nano
- Modules 里打开 tflite micro support
C Library
中选择Newlib C Library
Sub Systems and OS Services
中选中C++ support for the application
编写测试
主要功能是加载micro_speech.tflite
模型并识别一段语音是yes
orno
main.cpp 代码如下
注:GenerateMicroFeatures
无法链接成功,于是使用了预先生成好的 features。/* * Copyright (c) 2012-2014 Wind River Systems, Inc. * * SPDX-License-Identifier: Apache-2.0 */ #include <zephyr/zephyr.h> #include <tensorflow/lite/micro/all_ops_resolver.h> #include <tensorflow/lite/micro/micro_error_reporter.h> #include <tensorflow/lite/micro/micro_interpreter.h> #include <tensorflow/lite/schema/schema_generated.h> #include <tensorflow/lite/micro/testing/micro_test.h> #include "micro_features_generator.h" #include "micro_model_settings.h" #include "yes_micro_features_data.h" #define INCBIN_STYLE INCBIN_STYLE_SNAKE #include "incbin.h" INCBIN(micro_speech_tflite, "micro_speech.tflite"); const int tensor_arena_size = 64 * 1024; uint8_t tensor_arena[tensor_arena_size]; TF_LITE_MICRO_TESTS_BEGIN TF_LITE_MICRO_TEST(CSK6_TFLITE) { printk("Hello CSK6 TFLITE! %s\n", CONFIG_BOARD); micro_test::tests_passed = 0; micro_test::tests_failed = 0; tflite::InitializeTest(); tflite::MicroErrorReporter micro_error_reporter; tflite::ErrorReporter *error_reporter = µ_error_reporter; const tflite::Model *model = ::tflite::GetModel(gmicro_speech_tflite_data); if (model->version() != TFLITE_SCHEMA_VERSION) { TF_LITE_REPORT_ERROR(error_reporter, "Model provided is schema version %d not equal " "to supported version %d.\n", model->version(), TFLITE_SCHEMA_VERSION); } tflite::MicroMutableOpResolver<4> micro_op_resolver; micro_op_resolver.AddDepthwiseConv2D(); micro_op_resolver.AddFullyConnected(); micro_op_resolver.AddReshape(); micro_op_resolver.AddSoftmax(); tflite::MicroInterpreter interpreter(model, micro_op_resolver, tensor_arena, tensor_arena_size, error_reporter); interpreter.AllocateTensors(); TfLiteTensor *input = interpreter.input(0); TF_LITE_MICRO_EXPECT(input != nullptr); TF_LITE_MICRO_EXPECT_EQ(2, input->dims->size); TF_LITE_MICRO_EXPECT_EQ(1, input->dims->data[0]); TF_LITE_MICRO_EXPECT_EQ(1960, input->dims->data[1]); TF_LITE_MICRO_EXPECT_EQ(kTfLiteInt8, input->type); const int8_t *yes_features_data = g_yes_micro_f2e59fea_nohash_1_data; for (size_t i = 0; i < input->bytes; ++i) { input->data.int8[i] = yes_features_data[i]; } TfLiteStatus invoke_status = interpreter.Invoke(); if (invoke_status != kTfLiteOk) { MicroPrintf("Invoke failed\n"); } TF_LITE_MICRO_EXPECT_EQ(kTfLiteOk, invoke_status); const int kSilenceIndex = 0; const int kUnknownIndex = 1; const int kYesIndex = 2; const int kNoIndex = 3; TfLiteTensor *output = interpreter.output(0); TF_LITE_MICRO_EXPECT_EQ(2, output->dims->size); TF_LITE_MICRO_EXPECT_EQ(1, output->dims->data[0]); TF_LITE_MICRO_EXPECT_EQ(4, output->dims->data[1]); TF_LITE_MICRO_EXPECT_EQ(kTfLiteInt8, output->type); // Make sure that the expected "Yes" score is higher than the other classes. uint8_t silence_score = output->data.int8[kSilenceIndex] + 128; uint8_t unknown_score = output->data.int8[kUnknownIndex] + 128; uint8_t yes_score = output->data.int8[kYesIndex] + 128; uint8_t no_score = output->data.int8[kNoIndex] + 128; TF_LITE_MICRO_EXPECT_GT(yes_score, silence_score); TF_LITE_MICRO_EXPECT_GT(yes_score, unknown_score); TF_LITE_MICRO_EXPECT_GT(yes_score, no_score); } TF_LITE_MICRO_TESTS_END
烧录运行
(base) PS D:\Software\Development\listenai> .\cskburn -s \\.\COM5 -b 748800 -C 6 0x0 "X:\work\repo\csk6-learn\tflite_micro\build\zephyr\zephyr.bin" Partition 1: 0x00000000 (156.93 KB) - X:\work\repo\csk6-learn\tflite_micro\build\zephyr\zephyr.bin Waiting for device... Entering update mode... Detected flash size: 16 MB Burning partition 1/1... (0x00000000, 156.93 KB) 156.93 KB / 156.93 KB (100.00%) Finished
串口打印
**** Booting Zephyr OS build v1.1.1-alpha.2 *** Testing CSK6_TFLITE Hello CSK6 TFLITE! csk6011a_nano 1/1 tests passed ~~~ALL TESTS PASSED~~~
正确识别出了
yes
6. 碎碎念
希望官方快点放出 DSP 和 NPU 的工具链呀~