flutter metal实现图像处理详细流程-kb88凯时官网登录

背景

在之前自制的图像处理app中，使用了opengl处理图片，这次使用metal替代opengl，来达到更好的性能，顺便熟悉一下metal的渲染流程

基本思路

flutter使用cvpixelbuffer和ios交互，我们可以直接使用cvpixelbuffer创建mtltexture，然后将mtltexture设置为渲染目标。这样metal框架可以直接将渲染结果写入cvpixelbuffer，达到更加高效的目的。

metal环境设置

主要初始化device，pipelinestate，commandqueue三个对象。我们需要依赖device分配各种metal资源，pipelinestate管理着渲染流水线的各个环节的配置，比如vertex shader，fragment shader，输出像素格式等。commandqueue用于管理执行的绘制命令。

_device = mtlcreatesystemdefaultdevice();
id lib = [_device newdefaultlibrary];
id vertexfunc = [lib newfunctionwithname:vertexfuncname];
id fragfunc = [lib newfunctionwithname:fragfuncname];
mtlrenderpipelinedescriptor *renderpipelinedesc = [mtlrenderpipelinedescriptor new];
renderpipelinedesc.vertexfunction = vertexfunc;
renderpipelinedesc.fragmentfunction = fragfunc;
renderpipelinedesc.colorattachments[0].pixelformat = mtlpixelformatbgra8unorm;
_pipelinestate = [_device newrenderpipelinestatewithdescriptor:renderpipelinedesc error:nil];
_commandqueue = [_device newcommandqueue];

从cvpixelbuffer创建mtltexture纹理

首先创建一个cvpixelbuffer对象

nsdictionary *pixelattributes = @{( id )kcvpixelbufferiosurfacepropertieskey : @{}};
cvpixelbuffercreate(
            kcfallocatordefault,
                    imagewidth,
                    imageheight,
            kcvpixelformattype_32bgra,
                    (__bridge cfdictionaryref)pixelattributes,
                    &_rendertargetpixelbuffer);

利用cvmetaltexturecachecreatetexturefromimage从cvpixelbuffer创建mtltexture

cvreturn ret = cvmetaltexturecachecreate(kcfallocatordefault, nil, _mtcontext.device, nil, &_texturecache);
cvmetaltextureref rendertargetmetaltextureref;
ret = cvmetaltexturecachecreatetexturefromimage(kcfallocatordefault, _texturecache, _rendertargetpixelbuffer, nil, mtlpixelformatbgra8unorm, imagewidth, imageheight, 0, &rendertargetmetaltextureref);
id mtltexture = cvmetaltexturegettexture(rendertargetmetaltextureref);

渲染到纹理

从commandqueue获得一个commandbuffer，用于保存需要执行的绘制命令

_activecmdbuffer = [_commandqueue commandbuffer];

创建mtlrenderpassdescriptor设置本次绘制的相关配置，比如绘制到哪里，这里指定通过cvpixelbuffer创建出来的mtltexture，是否清除当前内容，清除的颜色

mtlrenderpassdescriptor *renderpassdesc = [mtlrenderpassdescriptor new];
renderpassdesc.colorattachments[0].texture = target;
renderpassdesc.colorattachments[0].loadaction = mtlloadactionclear;
renderpassdesc.colorattachments[0].clearcolor = mtlclearcolormake(0, 0, 0, 1);

通过commandbuffer和mtlrenderpassdescriptor创建一个mtlrendercommandencoder

_activeencoder = [_activecmdbuffer rendercommandencoderwithdescriptor:renderpassdesc];

指定mtlrendercommandencoder所在的pipelinestate

[_activeencoder setrenderpipelinestate:_pipelinestate];

使用mtlrendercommandencoder绑定buffer和texture，在metal里，uniform和vertex buffer 都是通过mtlbuffer绑定到shader中

[_activeencoder setvertexbuffer:vertexbuffer offset:0 atindex:0];
[_activeencoder setfragmentbuffer:uniformbuffer offset:0 atindex:0];
[_activeencoder setfragmentbuffer:texture offset:0 atindex:0];

绘制图形

[_activeencoder drawprimitives:mtlprimitivetypetriangle vertexstart:0 vertexcount:vertexcount instancecount:1];

显式的结束mtlrendercommandencoder

[_activeencoder endencoding];

提交commandbuffer

[_activecmdbuffer commit];

等待绘制结束，如果你想要异步等待，需要在[_activecmdbuffer commit]之前设置completedhandler

// 同步等待
[_activecmdbuffer waituntilcompleted];
// 异步等待
[_activecmdbuffer addcompletedhandler:^(id _nonnull buf) {
}];

到此绘制的内容就已经在cvpixelbuffer中了，再将cvpixelbuffer提交给flutter显示即可。