【融云分析】iOS 基于实时音视频 SDK 实现屏幕共享功能
Replaykit 介绍
在之前的 iOS 版本中，iOS 开发者只能拿到编码后的数据，拿不到原始的 PCM 和 YUV，到 iOS 10 之后，开发者可以拿到原始数据，但是只能录制 App 内的内容，如果切到后台，将停止录制，直到 iOS 11，苹果对屏幕共享进行了升级并开放了权限，既可以拿到原始数据，又可以录制整个系统，以下我们重点来说 iOS 11 之后的屏幕共享功能。
系统屏幕共享
- (void)initMode_1 {
self.systemBroadcastPickerView = [[RPSystemBroadcastPickerView alloc] initWithFrame:CGRectMake(0, 64, ScreenWidth, 80)];
self.systemBroadcastPickerView.preferredExtension = @"cn.rongcloud.replaytest.Recoder";
self.systemBroadcastPickerView.backgroundColor = [UIColor colorWithRed:53.0/255.0 green:129.0/255.0 blue:242.0/255.0 alpha:1.0];
self.systemBroadcastPickerView.showsMicrophoneButton = NO;
[self.view addSubview:self.systemBroadcastPickerView];
}
在 iOS 11 创建一个 Extension 之后，调用上面的代码就可以开启屏幕共享了，然后系统会为我们生成一个 SampleHandler 的类，在这个方法中，苹果会根据 RPSampleBufferType 上报不同类型的数据。
- (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer withType:(RPSampleBufferType)sampleBufferType
那怎么通过融云的 RongRTCLib 将屏幕共享数据发送出去呢？
1. 基于 Socket 的逼格玩法
1.1. Replaykit 框架启动和创建 Socket
//
// ViewController.m
// Socket_Replykit
//
// Created by Sun on 2020/5/19.
// Copyright © 2020 RongCloud. All rights reserved.
//
#import "ViewController.h"
#import <ReplayKit/ReplayKit.h>
#import "RongRTCServerSocket.h"
@interface ViewController ()<RongRTCServerSocketProtocol>
@property (nonatomic, strong) RPSystemBroadcastPickerView *systemBroadcastPickerView;
/**
server socket
*/
@property(nonatomic , strong)RongRTCServerSocket *serverSocket;
@end
@implementation ViewController
- (void)viewDidLoad {
[super viewDidLoad];
self.view.backgroundColor = [UIColor whiteColor];
// Do any additional setup after loading the view.
[self.serverSocket createServerSocket];
self.systemBroadcastPickerView = [[RPSystemBroadcastPickerView alloc] initWithFrame:CGRectMake(0, 64, [UIScreen mainScreen].bounds.size.width, 80)];
self.systemBroadcastPickerView.preferredExtension = @"cn.rongcloud.sealrtc.RongRTCRP";
self.systemBroadcastPickerView.backgroundColor = [UIColor colorWithRed:53.0/255.0 green:129.0/255.0 blue:242.0/255.0 alpha:1.0];
self.systemBroadcastPickerView.showsMicrophoneButton = NO;
[self.view addSubview:self.systemBroadcastPickerView];
}
- (RongRTCServerSocket *)serverSocket {
if (!_serverSocket) {
RongRTCServerSocket *socket = [[RongRTCServerSocket alloc] init];
socket.delegate = self;
_serverSocket = socket;
}
return _serverSocket;
}
- (void)didProcessSampleBuffer:(CMSampleBufferRef)sampleBuffer {
// 这里拿到了最终的数据，比如最后可以使用融云的音视频SDK RTCLib 进行传输就可以了
}
@end
其中，包括了创建 Server Socket 的步骤，我们把主 App 当做 Server，然后屏幕共享 Extension 当做 Client ，通过 Socket 向我们的主 APP 发送数据。
在 Extension 里面，我们拿到 ReplayKit 框架上报的屏幕视频数据后：
//
// SampleHandler.m
// SocketReply
//
// Created by Sun on 2020/5/19.
// Copyright © 2020 RongCloud. All rights reserved.
//
#import "SampleHandler.h"
#import "RongRTCClientSocket.h"
@interface SampleHandler()
/**
Client Socket
*/
@property (nonatomic, strong) RongRTCClientSocket *clientSocket;
@end
@implementation SampleHandler
- (void)broadcastStartedWithSetupInfo:(NSDictionary<NSString *,NSObject *> *)setupInfo {
// User has requested to start the broadcast. Setup info from the UI extension can be supplied but optional.
self.clientSocket = [[RongRTCClientSocket alloc] init];
[self.clientSocket createCliectSocket];
}
- (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer withType:(RPSampleBufferType)sampleBufferType {
switch (sampleBufferType) {
case RPSampleBufferTypeVideo:
// Handle video sample buffer
[self sendData:sampleBuffer];
break;
case RPSampleBufferTypeAudioApp:
// Handle audio sample buffer for app audio
break;
case RPSampleBufferTypeAudioMic:
// Handle audio sample buffer for mic audio
break;
default:
break;
}
}
- (void)sendData:(CMSampleBufferRef)sampleBuffer {
[self.clientSocket encodeBuffer:sampleBuffer];
}
@end
可见 ，这里我们创建了一个 Client Socket，然后拿到屏幕共享的视频 sampleBuffer 之后，通过 Socket 发给我们的主 App，这就是屏幕共享的流程。
1.2 Local Socket 的使用
//
// RongRTCSocket.m
// SealRTC
//
// Created by Sun on 2020/5/7.
// Copyright © 2020 RongCloud. All rights reserved.
//
#import "RongRTCSocket.h"
#import <arpa/inet.h>
#import <netdb.h>
#import <sys/types.h>
#import <sys/socket.h>
#import <ifaddrs.h>
#import "RongRTCThread.h"
@interface RongRTCSocket()
/**
receive thread
*/
@property (nonatomic, strong) RongRTCThread *receiveThread;
@end
@implementation RongRTCSocket
- (int)createSocket {
int socket = socket(AF_INET, SOCK_STREAM, 0);
self.socket = socket;
if (self.socket == -1) {
close(self.socket);
NSLog(@"socket error : %d", self.socket);
}
self.receiveThread = [[RongRTCThread alloc] init];
[self.receiveThread run];
return socket;
}
- (void)setSendBuffer {
int optVal = 1024 * 1024 * 2;
int optLen = sizeof(int);
int res = setsockopt(self.socket, SOL_SOCKET, SO_SNDBUF, (char *)&optVal,optLen);
NSLog(@"set send buffer:%d", res);
}
- (void)setReceiveBuffer {
int optVal = 1024 * 1024 * 2;
int optLen = sizeof(int);
int res = setsockopt(self.socket, SOL_SOCKET, SO_RCVBUF, (char*)&optVal,optLen );
NSLog(@"set send buffer:%d",res);
}
- (void)setSendingTimeout {
struct timeval timeout = {10,0};
int res = setsockopt(self.socket, SOL_SOCKET, SO_SNDTIMEO, (char *)&timeout, sizeof(int));
NSLog(@"set send timeout:%d", res);
}
- (void)setReceiveTimeout {
struct timeval timeout = {10, 0};
int res = setsockopt(self.socket, SOL_SOCKET, SO_RCVTIMEO, (char *)&timeout, sizeof(int));
NSLog(@"set send timeout:%d", res);
}
- (BOOL)connect {
NSString *serverHost = [self ip];
struct hostent *server = gethostbyname([serverHost UTF8String]);
if (server == NULL) {
close(self.socket);
NSLog(@"get host error");
return NO;
}
struct in_addr *remoteAddr = (struct in_addr *)server->h_addr_list[0];
struct sockaddr_in addr;
addr.sin_family = AF_INET;
addr.sin_addr = *remoteAddr;
addr.sin_port = htons(CONNECTPORT);
int res = connect(self.socket, (struct sockaddr *) &addr, sizeof(addr));
if (res == -1) {
close(self.socket);
NSLog(@"connect error");
return NO;
}
NSLog(@"socket connect to server success");
return YES;
}
- (BOOL)bind {
struct sockaddr_in client;
client.sin_family = AF_INET;
NSString *ipStr = [self ip];
if (ipStr.length <= 0) {
return NO;
}
const char *ip = [ipStr cStringUsingEncoding:NSASCIIStringEncoding];
client.sin_addr.s_addr = inet_addr(ip);
client.sin_port = htons(CONNECTPORT);
int bd = bind(self.socket, (struct sockaddr *) &client, sizeof(client));
if (bd == -1) {
close(self.socket);
NSLog(@"bind error: %d", bd);
return NO;
}
return YES;
}
- (BOOL)listen {
int ls = listen(self.socket, 128);
if (ls == -1) {
close(self.socket);
NSLog(@"listen error: %d", ls);
return NO;
}
return YES;
}
- (void)receive {
dispatch_async(dispatch_get_global_queue(0, 0), ^{
[self receiveData];
});
}
- (NSString *)ip {
NSString *ip = nil;
struct ifaddrs *addrs = NULL;
struct ifaddrs *tmpAddrs = NULL;
BOOL res = getifaddrs(&addrs);
if (res == 0) {
tmpAddrs = addrs;
while (tmpAddrs != NULL) {
if (tmpAddrs->ifa_addr->sa_family == AF_INET) {
// Check if interface is en0 which is the wifi connection on the iPhone
NSLog(@"%@", [NSString stringWithUTF8String:inet_ntoa(((struct sockaddr_in *)tmpAddrs->ifa_addr)->sin_addr)]);
if ([[NSString stringWithUTF8String:tmpAddrs->ifa_name] isEqualToString:@"en0"]) {
// Get NSString from C String
ip = [NSString stringWithUTF8String:inet_ntoa(((struct sockaddr_in *)tmpAddrs->ifa_addr)->sin_addr)];
}
}
tmpAddrs = tmpAddrs->ifa_next;
}
}
// Free memory
freeifaddrs(addrs);
NSLog(@"%@",ip);
return ip;
}
- (void)close {
int res = close(self.socket);
NSLog(@"shut down: %d", res);
}
- (void)receiveData {
}
- (void)dealloc {
[self.receiveThread stop];
}
@end
首先创建了一个 Socket 的父类，然后用 Server Socket 和 Client Socket 分别继承类来实现链接、绑定等操作。可以看到有些数据可以设置，有些则不用，这里不是核心，核心是怎样收发数据。
1.3 发送屏幕共享数据
//
// RongRTCClientSocket.m
// SealRTC
//
// Created by Sun on 2020/5/7.
// Copyright © 2020 RongCloud. All rights reserved.
//
#import "RongRTCClientSocket.h"
#import <arpa/inet.h>
#import <netdb.h>
#import <sys/types.h>
#import <sys/socket.h>
#import <ifaddrs.h>
#import "RongRTCThread.h"
#import "RongRTCSocketHeader.h"
#import "RongRTCVideoEncoder.h"
@interface RongRTCClientSocket() <RongRTCCodecProtocol> {
pthread_mutex_t lock;
}
/**
video encoder
*/
@property (nonatomic, strong) RongRTCVideoEncoder *encoder;
/**
encode queue
*/
@property (nonatomic, strong) dispatch_queue_t encodeQueue;
@end
@implementation RongRTCClientSocket
- (BOOL)createClientSocket {
if ([self createSocket] == -1) {
return NO;
}
BOOL isC = [self connect];
[self setSendBuffer];
[self setSendingTimeout];
if (isC) {
_encodeQueue = dispatch_queue_create("cn.rongcloud.encodequeue", NULL);
[self createVideoEncoder];
return YES;
} else {
return NO;
}
}
- (void)createVideoEncoder {
self.encoder = [[RongRTCVideoEncoder alloc] init];
self.encoder.delegate = self;
RongRTCVideoEncoderSettings *settings = [[RongRTCVideoEncoderSettings alloc] init];
settings.width = 720;
settings.height = 1280;
settings.startBitrate = 300;
settings.maxFramerate = 30;
settings.minBitrate = 1000;
[self.encoder configWithSettings:settings onQueue:_encodeQueue];
}
- (void)clientSend:(NSData *)data {
//data length
NSUInteger dataLength = data.length;
// data header struct
DataHeader dataH;
memset((void *)&dataH, 0, sizeof(dataH));
// pre
PreHeader preH;
memset((void *)&preH, 0, sizeof(preH));
preH.pre[0] = '&';
preH.dataLength = dataLength;
dataH.preH = preH;
// buffer
int headerlength = sizeof(dataH);
int totalLength = dataLength + headerlength;
// srcbuffer
Byte *src = (Byte *)[data bytes];
// send buffer
char *buffer = (char *)malloc(totalLength * sizeof(char));
memcpy(buffer, &dataH, headerlength);
memcpy(buffer + headerlength, src, dataLength);
// tosend
[self sendBytes:buffer length:totalLength];
free(buffer);
}
- (void)encodeBuffer:(CMSampleBufferRef)sampleBuffer {
[self.encoder encode:sampleBuffer];
}
- (void)sendBytes:(char *)bytes length:(int)length {
LOCK(self->lock);
int hasSendLength = 0;
while (hasSendLength < length) {
// connect socket success
if (self.socket > 0) {
// send
int sendRes = send(self.socket, bytes, length - hasSendLength, 0);
if (sendRes == -1 || sendRes == 0) {
UNLOCK(self->lock);
NSLog(@"send buffer error");
[self close];
break;
}
hasSendLength += sendRes;
bytes += sendRes;
} else {
NSLog(@"client socket connect error");
UNLOCK(self->lock);
}
}
UNLOCK(self->lock);
}
- (void)spsData:(NSData *)sps ppsData:(NSData *)pps {
[self clientSend:sps];
[self clientSend:pps];
}
- (void)naluData:(NSData *)naluData {
[self clientSend:naluData];
}
- (void)deallo c{
NSLog(@"dealoc cliect socket");
}
@end
这里的核心思想是拿到屏幕共享的数据之后，先进行压缩，当压缩完成后会通过回调上报给当前类。既而通过 clientSend 方法，发给主 App。发给主 App 的数据中自定义了一个头部，头部添加了一个前缀和一个每次发送字节的长度，当接收端收到数据包后解析即可。
- (void)clientSend:(NSData *)data {
//data length
NSUInteger dataLength = data.length;
// data header struct
DataHeader dataH;
memset((void *)&dataH, 0, sizeof(dataH));
// pre
PreHeader preH;
memset((void *)&preH, 0, sizeof(preH));
preH.pre[0] = '&';
preH.dataLength = dataLength;
dataH.preH = preH;
// buffer
int headerlength = sizeof(dataH);
int totalLength = dataLength + headerlength;
// srcbuffer
Byte *src = (Byte *)[data bytes];
// send buffer
char *buffer = (char *)malloc(totalLength * sizeof(char));
memcpy(buffer, &dataH, headerlength);
memcpy(buffer + headerlength, src, dataLength);
// to send
[self sendBytes:buffer length:totalLength];
free(buffer);
}
1.4 接收屏幕共享数据
//
// RongRTCServerSocket.m
// SealRTC
//
// Created by Sun on 2020/5/7.
// Copyright © 2020 RongCloud. All rights reserved.
//
#import "RongRTCServerSocket.h"
#import <arpa/inet.h>
#import <netdb.h>
#import <sys/types.h>
#import <sys/socket.h>
#import <ifaddrs.h>
#import <UIKit/UIKit.h>
#import "RongRTCThread.h"
#import "RongRTCSocketHeader.h"
#import "RongRTCVideoDecoder.h"
@interface RongRTCServerSocket() <RongRTCCodecProtocol>
{
pthread_mutex_t lock;
int _frameTime;
CMTime _lastPresentationTime;
Float64 _currentMediaTime;
Float64 _currentVideoTime;
dispatch_queue_t _frameQueue;
}
@property (nonatomic, assign) int acceptSocket;
/**
data length
*/
@property (nonatomic, assign) NSUInteger dataLength;
/**
timeData
*/
@property (nonatomic, strong) NSData *timeData;
/**
decoder queue
*/
@property (nonatomic, strong) dispatch_queue_t decoderQueue;
/**
decoder
*/
@property (nonatomic, strong) RongRTCVideoDecoder *decoder;
@end
@implementation RongRTCServerSocket
- (BOOL)createServerSocket {
if ([self createSocket] == -1) {
return NO;
}
[self setReceiveBuffer];
[self setReceiveTimeout];
BOOL isB = [self bind];
BOOL isL = [self listen];
if (isB && isL) {
_decoderQueue = dispatch_queue_create("cn.rongcloud.decoderQueue", NULL);
_frameTime = 0;
[self createDecoder];
[self receive];
return YES;
} else {
return NO;
}
}
- (void)createDecoder {
self.decoder = [[RongRTCVideoDecoder alloc] init];
self.decoder.delegate = self;
RongRTCVideoEncoderSettings *settings = [[RongRTCVideoEncoderSettings alloc] init];
settings.width = 720;
settings.height = 1280;
settings.startBitrate = 300;
settings.maxFramerate = 30;
settings.minBitrate = 1000;
[self.decoder configWithSettings:settings onQueue:_decoderQueue];
}
- (void)receiveData {
struct sockaddr_in rest;
socklen_t rest_size = sizeof(struct sockaddr_in);
self.acceptSocket = accept(self.socket, (struct sockaddr *) &rest, &rest_size);
while (self.acceptSocket != -1) {
DataHeader dataH;
memset(&dataH, 0, sizeof(dataH));
if (![self receiveData:(char *)&dataH length:sizeof(dataH)]) {
continue;
}
PreHeader preH = dataH.preH;
char pre = preH.pre[0];
if (pre == '&') {
// rongcloud socket
NSUInteger dataLenght = preH.dataLength;
char *buff = (char *)malloc(sizeof(char) * dataLenght);
if ([self receiveData:(char *)buff length:dataLenght]) {
NSData *data = [NSData dataWithBytes:buff length:dataLenght];
[self.decoder decode:data];
free(buff);
}
} else {
NSLog(@"pre is not &");
return;
}
}
}
- (BOOL)receiveData:(char *)data length:(NSUInteger)length {
LOCK(lock);
int receiveLength = 0;
while (receiveLength < length) {
ssize_t res = recv(self.acceptSocket, data, length - receiveLength, 0);
if (res == -1 || res == 0) {
UNLOCK(lock);
NSLog(@"receive data error");
break;
}
receiveLength += res;
data += res;
}
UNLOCK(lock);
return YES;
}
- (void)didGetDecodeBuffer:(CVPixelBufferRef)pixelBuffer {
_frameTime += 1000;
CMTime pts = CMTimeMake(_frameTime, 1000);
CMSampleBufferRef sampleBuffer = [RongRTCBufferUtil sampleBufferFromPixbuffer:pixelBuffer time:pts];
// Check to see if there is a problem with the decoded data. If the image appears, you are right.
UIImage *image = [RongRTCBufferUtil imageFromBuffer:sampleBuffer];
[self.delegate didProcessSampleBuffer:sampleBuffer];
CFRelease(sampleBuffer);
}
- (void)close {
int res = close(self.acceptSocket);
self.acceptSocket = -1;
NSLog(@"shut down server: %d", res);
[super close];
}
- (void)dealloc {
NSLog(@"dealoc server socket");
}
@end
主 App 通过 Socket 会持续收到数据包，再将数据包进行解码，将解码后的数据通过代理 didGetDecodeBuffer 代理方法回调给 App 层。App 层就可以通过融云 RongRTCLib 的发送自定义流方法将视频数据发送到对端。
1.5 VideotoolBox 硬编码
//
// RongRTCVideoEncoder.m
// SealRTC
//
// Created by Sun on 2020/5/13.
// Copyright © 2020 RongCloud. All rights reserved.
//
#import "RongRTCVideoEncoder.h"
#import "helpers.h"
@interface RongRTCVideoEncoder() {
VTCompressionSessionRef _compressionSession;
int _frameTime;
}
/**
settings
*/
@property (nonatomic, strong) RongRTCVideoEncoderSettings *settings;
/**
callback queue
*/
@property (nonatomic , strong ) dispatch_queue_t callbackQueue;
- (void)sendSpsAndPPSWithSampleBuffer:(CMSampleBufferRef)sampleBuffer;
- (void)sendNaluData:(CMSampleBufferRef)sampleBuffer;
@end
void compressionOutputCallback(void *encoder,
void *params,
OSStatus status,
VTEncodeInfoFlags infoFlags,
CMSampleBufferRef sampleBuffer) {
RongRTCVideoEncoder *videoEncoder = (__bridge RongRTCVideoEncoder *)encoder;
if (status != noErr) {
return;
}
if (infoFlags & kVTEncodeInfo_FrameDropped) {
return;
}
BOOL isKeyFrame = NO;
CFArrayRef attachments = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, 0);
if (attachments != nullptr && CFArrayGetCount(attachments)) {
CFDictionaryRef attachment = static_cast<CFDictionaryRef>(CFArrayGetValueAtIndex(attachments, 0)) ;
isKeyFrame = !CFDictionaryContainsKey(attachment, kCMSampleAttachmentKey_NotSync);
}
CMBlockBufferRef block_buffer = CMSampleBufferGetDataBuffer(sampleBuffer);
CMBlockBufferRef contiguous_buffer = nullptr;
if (!CMBlockBufferIsRangeContiguous(block_buffer, 0, 0)) {
status = CMBlockBufferCreateContiguous(nullptr, block_buffer, nullptr, nullptr, 0, 0, 0, &contiguous_buffer);
if (status != noErr) {
return;
}
} else {
contiguous_buffer = block_buffer;
CFRetain(contiguous_buffer);
block_buffer = nullptr;
}
size_t block_buffer_size = CMBlockBufferGetDataLength(contiguous_buffer);
if (isKeyFrame) {
[videoEncoder sendSpsAndPPSWithSampleBuffer:sampleBuffer];
}
if (contiguous_buffer) {
CFRelease(contiguous_buffer);
}
[videoEncoder sendNaluData:sampleBuffer];
}
@implementation RongRTCVideoEncoder
@synthesize settings = _settings;
@synthesize callbackQueue = _callbackQueue;
- (BOOL)configWithSettings:(RongRTCVideoEncoderSettings *)settings onQueue:(nonnull dispatch_queue_t)queue {
self.settings = settings;
if (queue) {
_callbackQueue = queue;
} else {
_callbackQueue = dispatch_get_main_queue();
}
if ([self resetCompressionSession:settings]) {
_frameTime = 0;
return YES;
} else {
return NO;
}
}
- (BOOL)resetCompressionSession:(RongRTCVideoEncoderSettings *)settings {
[self destroyCompressionSession];
OSStatus status = VTCompressionSessionCreate(nullptr, settings.width, settings.height, kCMVideoCodecType_H264, nullptr, nullptr, nullptr, compressionOutputCallback, (__bridge void * _Nullable)(self), &_compressionSession);
if (status != noErr) {
return NO;
}
[self configureCompressionSession:settings];
return YES;
}
- (void)configureCompressionSession:(RongRTCVideoEncoderSettings *)settings {
if (_compressionSession) {
SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_RealTime, true);
SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_ProfileLevel, kVTProfileLevel_H264_Baseline_AutoLevel);
SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_AllowFrameReordering, false);
SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_MaxKeyFrameInterval, 10);
uint32_t targetBps = settings.startBitrate * 1000;
SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_AverageBitRate, targetBps);
SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_ExpectedFrameRate, settings.maxFramerate);
int bitRate = settings.width * settings.height * 3 * 4 * 4;
SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_AverageBitRate, bitRate);
int bitRateLimit = settings.width * settings.height * 3 * 4;
SetVTSessionProperty(_compressionSession, kVTCompressionPropertyKey_DataRateLimits, bitRateLimit);
}
}
- (void)encode:(CMSampleBufferRef)sampleBuffer {
CVImageBufferRef imageBuffer = (CVImageBufferRef)CMSampleBufferGetImageBuffer(sampleBuffer);
CMTime pts = CMTimeMake(self->_frameTime++, 1000);
VTEncodeInfoFlags flags;
OSStatus res = VTCompressionSessionEncodeFrame(self->_compressionSession,
imageBuffer,
pts,
kCMTimeInvalid,
NULL, NULL, &flags);
if (res != noErr) {
NSLog(@"encode frame error:%d", (int)res);
VTCompressionSessionInvalidate(self->_compressionSession);
CFRelease(self->_compressionSession);
self->_compressionSession = NULL;
return;
}
}
- (void)sendSpsAndPPSWithSampleBuffer:(CMSampleBufferRef)sampleBuffer {
CMFormatDescriptionRef format = CMSampleBufferGetFormatDescription(sampleBuffer);
const uint8_t *sps ;
const uint8_t *pps;
size_t spsSize ,ppsSize , spsCount,ppsCount;
OSStatus spsStatus = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(format, 0, &sps, &spsSize, &spsCount, NULL);
OSStatus ppsStatus = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(format, 1, &pps, &ppsSize, &ppsCount, NULL);
if (spsStatus == noErr && ppsStatus == noErr) {
const char bytes[] = "\x00\x00\x00\x01";
size_t length = (sizeof bytes) - 1;
NSMutableData *spsData = [NSMutableData dataWithCapacity:4+ spsSize];
NSMutableData *ppsData = [NSMutableData dataWithCapacity:4 + ppsSize];
[spsData appendBytes:bytes length:length];
[spsData appendBytes:sps length:spsSize];
[ppsData appendBytes:bytes length:length];
[ppsData appendBytes:pps length:ppsSize];
if (self && self.callbackQueue) {
dispatch_async(self.callbackQueue, ^{
if (self.delegate && [self.delegate respondsToSelector:@selector(spsData:ppsData:)]) {
[self.delegate spsData:spsData ppsData:ppsData];
}
});
}
} else {
NSLog(@"sps status:%@, pps status:%@", @(spsStatus), @(ppsStatus));
}
}
- (void)sendNaluData:(CMSampleBufferRef)sampleBuffer {
size_t totalLength = 0;
size_t lengthAtOffset=0;
char *dataPointer;
CMBlockBufferRef blockBuffer = CMSampleBufferGetDataBuffer(sampleBuffer);
OSStatus status1 = CMBlockBufferGetDataPointer(blockBuffer, 0, &lengthAtOffset, &totalLength, &dataPointer);
if (status1 != noErr) {
NSLog(@"video encoder error, status = %d", (int)status1);
return;
}
static const int h264HeaderLength = 4;
size_t bufferOffset = 0;
while (bufferOffset < totalLength - h264HeaderLength) {
uint32_t naluLength = 0;
memcpy(&naluLength, dataPointer + bufferOffset, h264HeaderLength);
naluLength = CFSwapInt32BigToHost(naluLength);
const char bytes[] = "\x00\x00\x00\x01";
NSMutableData *naluData = [NSMutableData dataWithCapacity:4 + naluLength];
[naluData appendBytes:bytes length:4];
[naluData appendBytes:dataPointer + bufferOffset + h264HeaderLength length:naluLength];
dispatch_async(self.callbackQueue, ^{
if (self.delegate && [self.delegate respondsToSelector:@selector(naluData:)]) {
[self.delegate naluData:naluData];
}
});
bufferOffset += naluLength + h264HeaderLength;
}
}
- (void)destroyCompressionSession {
if (_compressionSession) {
VTCompressionSessionInvalidate(_compressionSession);
CFRelease(_compressionSession);
_compressionSession = nullptr;
}
}
- (void)dealloc {
if (_compressionSession) {
VTCompressionSessionCompleteFrames(_compressionSession, kCMTimeInvalid);
VTCompressionSessionInvalidate(_compressionSession);
CFRelease(_compressionSession);
_compressionSession = NULL;
}
}
@end
1.6 VideotoolBox 解码
//
// RongRTCVideoDecoder.m
// SealRTC
//
// Created by Sun on 2020/5/14.
// Copyright © 2020 RongCloud. All rights reserved.
//
#import "RongRTCVideoDecoder.h"
#import <UIKit/UIKit.h>
#import "helpers.h"
@interface RongRTCVideoDecoder() {
uint8_t *_sps;
NSUInteger _spsSize;
uint8_t *_pps;
NSUInteger _ppsSize;
CMVideoFormatDescriptionRef _videoFormatDescription;
VTDecompressionSessionRef _decompressionSession;
}
/**
settings
*/
@property (nonatomic, strong) RongRTCVideoEncoderSettings *settings;
/**
callback queue
*/
@property (nonatomic, strong) dispatch_queue_t callbackQueue;
@end
void DecoderOutputCallback(void * CM_NULLABLE decompressionOutputRefCon,
void * CM_NULLABLE sourceFrameRefCon,
OSStatus status,
VTDecodeInfoFlags infoFlags,
CM_NULLABLE CVImageBufferRef imageBuffer,
CMTime presentationTimeStamp,
CMTime presentationDuration ) {
if (status != noErr) {
NSLog(@" decoder callback error :%@", @(status));
return;
}
CVPixelBufferRef *outputPixelBuffer = (CVPixelBufferRef *)sourceFrameRefCon;
*outputPixelBuffer = CVPixelBufferRetain(imageBuffer);
RongRTCVideoDecoder *decoder = (__bridge RongRTCVideoDecoder *)(decompressionOutputRefCon);
dispatch_async(decoder.callbackQueue, ^{
[decoder.delegate didGetDecodeBuffer:imageBuffer];
CVPixelBufferRelease(imageBuffer);
});
}
@implementation RongRTCVideoDecoder
@synthesize settings = _settings;
@synthesize callbackQueue = _callbackQueue;
- (BOOL)configWithSettings:(RongRTCVideoEncoderSettings *)settings onQueue:(dispatch_queue_t)queue {
self.settings = settings;
if (queue) {
_callbackQueue = queue;
} else {
_callbackQueue = dispatch_get_main_queue();
}
return YES;
}
- (BOOL)createVT {
if (_decompressionSession) {
return YES;
}
const uint8_t * const parameterSetPointers[2] = {_sps, _pps};
const size_t parameterSetSizes[2] = {_spsSize, _ppsSize};
int naluHeaderLen = 4;
OSStatus status = CMVideoFormatDescriptionCreateFromH264ParameterSets(kCFAllocatorDefault, 2, parameterSetPointers, parameterSetSizes, naluHeaderLen, &_videoFormatDescription );
if (status != noErr) {
NSLog(@"CMVideoFormatDescriptionCreateFromH264ParameterSets error:%@", @(status));
return false;
}
NSDictionary *destinationImageBufferAttributes =
@{
(id)kCVPixelBufferPixelFormatTypeKey: [NSNumber numberWithInt:kCVPixelFormatType_420YpCbCr8BiPlanarFullRange],
(id)kCVPixelBufferWidthKey: [NSNumber numberWithInteger:self.settings.width],
(id)kCVPixelBufferHeightKey: [NSNumber numberWithInteger:self.settings.height],
(id)kCVPixelBufferOpenGLCompatibilityKey: [NSNumber numberWithBool:true]
};
VTDecompressionOutputCallbackRecord CallBack;
CallBack.decompressionOutputCallback = DecoderOutputCallback;
CallBack.decompressionOutputRefCon = (__bridge void * _Nullable)(self);
status = VTDecompressionSessionCreate(kCFAllocatorDefault, _videoFormatDescription, NULL, (__bridge CFDictionaryRef _Nullable)(destinationImageBufferAttributes), &CallBack, &_decompressionSession);
if (status != noErr) {
NSLog(@"VTDecompressionSessionCreate error:%@", @(status));
return false;
}
status = VTSessionSetProperty(_decompressionSession, kVTDecompressionPropertyKey_RealTime,kCFBooleanTrue);
return YES;
}
- (CVPixelBufferRef)decode:(uint8_t *)frame withSize:(uint32_t)frameSize {
CVPixelBufferRef outputPixelBuffer = NULL;
CMBlockBufferRef blockBuffer = NULL;
CMBlockBufferFlags flag0 = 0;
OSStatus status = CMBlockBufferCreateWithMemoryBlock(kCFAllocatorDefault, frame, frameSize, kCFAllocatorNull, NULL, 0, frameSize, flag0, &blockBuffer);
if (status != kCMBlockBufferNoErr) {
NSLog(@"VCMBlockBufferCreateWithMemoryBlock code=%d", (int)status);
CFRelease(blockBuffer);
return outputPixelBuffer;
}
CMSampleBufferRef sampleBuffer = NULL;
const size_t sampleSizeArray[] = {frameSize};
status = CMSampleBufferCreateReady(kCFAllocatorDefault, blockBuffer, _videoFormatDescription, 1, 0, NULL, 1, sampleSizeArray, &sampleBuffer);
if (status != noErr || !sampleBuffer) {
NSLog(@"CMSampleBufferCreateReady failed status=%d", (int)status);
CFRelease(blockBuffer);
return outputPixelBuffer;
}
VTDecodeFrameFlags flag1 = kVTDecodeFrame_1xRealTimePlayback;
VTDecodeInfoFlags infoFlag = kVTDecodeInfo_Asynchronous;
status = VTDecompressionSessionDecodeFrame(_decompressionSession, sampleBuffer, flag1, &outputPixelBuffer, &infoFlag);
if (status == kVTInvalidSessionErr) {
NSLog(@"decode frame error with session err status =%d", (int)status);
[self resetVT];
} else {
if (status != noErr) {
NSLog(@"decode frame error with status =%d", (int)status);
}
}
CFRelease(sampleBuffer);
CFRelease(blockBuffer);
return outputPixelBuffer;
}
- (void)resetVT {
[self destorySession];
[self createVT];
}
- (void)decode:(NSData *)data {
uint8_t *frame = (uint8_t*)[data bytes];
uint32_t length = data.length;
uint32_t nalSize = (uint32_t)(length - 4);
uint32_t *pNalSize = (uint32_t *)frame;
*pNalSize = CFSwapInt32HostToBig(nalSize);
int type = (frame[4] & 0x1F);
CVPixelBufferRef pixelBuffer = NULL;
switch (type) {
case 0x05:
if ([self createVT]) {
pixelBuffer= [self decode:frame withSize:length];
}
break;
case 0x07:
self->_spsSize = length - 4;
self->_sps = (uint8_t *)malloc(self->_spsSize);
memcpy(self->_sps, &frame[4], self->_spsSize);
break;
case 0x08:
self->_ppsSize = length - 4;
self->_pps = (uint8_t *)malloc(self->_ppsSize);
memcpy(self->_pps, &frame[4], self->_ppsSize);
break;
default:
if ([self createVT]) {
pixelBuffer = [self decode:frame withSize:length];
}
break;
}
}
- (void)dealloc {
[self destorySession];
}
- (void)destorySession {
if (_decompressionSession) {
VTDecompressionSessionInvalidate(_decompressionSession);
CFRelease(_decompressionSession);
_decompressionSession = NULL;
}
}
@end
1.7 工具类
//
// RongRTCBufferUtil.m
// SealRTC
//
// Created by Sun on 2020/5/8.
// Copyright © 2020 RongCloud. All rights reserved.
//
#import "RongRTCBufferUtil.h"
// 下面的这些方法，一定要记得release，有的没有在方法里面release，但是在外面release了，要不然会内存泄漏
@implementation RongRTCBufferUtil
+ (UIImage *)imageFromBuffer:(CMSampleBufferRef)buffer {
CVPixelBufferRef pixelBuffer = (CVPixelBufferRef)CMSampleBufferGetImageBuffer(buffer);
CIImage *ciImage = [CIImage imageWithCVPixelBuffer:pixelBuffer];
CIContext *temporaryContext = [CIContext contextWithOptions:nil];
CGImageRef videoImage = [temporaryContext createCGImage:ciImage fromRect:CGRectMake(0, 0, CVPixelBufferGetWidth(pixelBuffer), CVPixelBufferGetHeight(pixelBuffer))];
UIImage *image = [UIImage imageWithCGImage:videoImage];
CGImageRelease(videoImage);
return image;
}
+ (UIImage *)compressImage:(UIImage *)image newWidth:(CGFloat)newImageWidth {
if (!image) return nil;
float imageWidth = image.size.width;
float imageHeight = image.size.height;
float width = newImageWidth;
float height = image.size.height/(image.size.width/width);
float widthScale = imageWidth /width;
float heightScale = imageHeight /height;
UIGraphicsBeginImageContext(CGSizeMake(width, height));
if (widthScale > heightScale) {
[image drawInRect:CGRectMake(0, 0, imageWidth /heightScale , height)];
}
else {
[image drawInRect:CGRectMake(0, 0, width , imageHeight /widthScale)];
}
UIImage *newImage = UIGraphicsGetImageFromCurrentImageContext();
UIGraphicsEndImageContext();
return newImage;
}
+ (CVPixelBufferRef)CVPixelBufferRefFromUiImage:(UIImage *)img {
CGSize size = img.size;
CGImageRef image = [img CGImage];
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithBool:YES], kCVPixelBufferCGImageCompatibilityKey,
[NSNumber numberWithBool:YES], kCVPixelBufferCGBitmapContextCompatibilityKey, nil];
CVPixelBufferRef pxbuffer = NULL;
CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault, size.width, size.height, kCVPixelFormatType_32ARGB, (__bridge CFDictionaryRef) options, &pxbuffer);
NSParameterAssert(status == kCVReturnSuccess && pxbuffer != NULL);
CVPixelBufferLockBaseAddress(pxbuffer, 0);
void *pxdata = CVPixelBufferGetBaseAddress(pxbuffer);
NSParameterAssert(pxdata != NULL);
CGColorSpaceRef rgbColorSpace = CGColorSpaceCreateDeviceRGB();
CGContextRef context = CGBitmapContextCreate(pxdata, size.width, size.height, 8, 4*size.width, rgbColorSpace, kCGImageAlphaPremultipliedFirst);
NSParameterAssert(context);
CGContextDrawImage(context, CGRectMake(0, 0, CGImageGetWidth(image), CGImageGetHeight(image)), image);
CGColorSpaceRelease(rgbColorSpace);
CGContextRelease(context);
CVPixelBufferUnlockBaseAddress(pxbuffer, 0);
return pxbuffer;
}
+ (CMSampleBufferRef)sampleBufferFromPixbuffer:(CVPixelBufferRef)pixbuffer time:(CMTime)time {
CMSampleBufferRef sampleBuffer = NULL;
//获取视频信息
CMVideoFormatDescriptionRef videoInfo = NULL;
OSStatus result = CMVideoFormatDescriptionCreateForImageBuffer(NULL, pixbuffer, &videoInfo);
CMTime currentTime = time;
// CMSampleTimingInfo timing = {currentTime, currentTime, kCMTimeInvalid};
CMSampleTimingInfo timing = {currentTime, currentTime, kCMTimeInvalid};
result = CMSampleBufferCreateForImageBuffer(kCFAllocatorDefault,pixbuffer, true, NULL, NULL, videoInfo, &timing, &sampleBuffer);
CFRelease(videoInfo);
return sampleBuffer;
}
+ (size_t)getCMTimeSize {
size_t size = sizeof(CMTime);
return size;
}
@end
此工具类中实现是由 CPU 处理，当进行 CMSampleBufferRef 转 UIImage、UIImage 转 CVPixelBufferRef、 CVPixelBufferRef 转 CMSampleBufferRef 以及裁剪图片时，这里需要注意将使用后的对象及时释放，否则会出现内存大量泄漏。
2. 视频发送
2.1 准备阶段
使用融云的 RongRTCLib 的前提需要一个 AppKey，请在官网（https://www.rongcloud.cn/)获取，通过 AppKey 取得 token 之后进行 IM 连接，在连接成功后加入 RTC 房间，这是屏幕共享发送的准备阶段。
- (void)broadcastStartedWithSetupInfo:(NSDictionary<NSString *,NSObject *> *)setupInfo {
// User has requested to start the broadcast. Setup info from the UI extension can be supplied but optional.
// 请填写您的 AppKey
self.appKey = @"";
// 请填写用户的 Token
self.token = @"";
// 请指定房间号
self.roomId = @"123456";
[[RCIMClient sharedRCIMClient] initWithAppKey:self.appKey];
[[RCIMClient sharedRCIMClient] setLogLevel:RC_Log_Level_Verbose];
// 连接 IM
[[RCIMClient sharedRCIMClient] connectWithToken:self.token
dbOpened:^(RCDBErrorCode code) {
NSLog(@"dbOpened: %zd", code);
} success:^(NSString *userId) {
NSLog(@"connectWithToken success userId: %@", userId);
// 加入房间
[[RCRTCEngine sharedInstance] joinRoom:self.roomId
completion:^(RCRTCRoom * _Nullable room, RCRTCCode code) {
self.room = room;
self.room.delegate = self;
[self publishScreenStream];
}];
} error:^(RCConnectErrorCode errorCode) {
NSLog(@"ERROR status: %zd", errorCode);
}];
}
如上是连接 IM 和加入 RTC 房间的全过程，其中还包含调用发布自定义视频 [self publishScreenStream]; 此方法在加入房间成功后才可以进行。
- (void)publishScreenStream {
RongRTCStreamParams *param = [[RongRTCStreamParams alloc] init];
param.videoSizePreset = RongRTCVideoSizePreset1280x720;
self.videoOutputStream = [[RongRTCAVOutputStream alloc] initWithParameters:param tag:@"RongRTCScreenVideo"];
[self.room publishAVStream:self.videoOutputStream extra:@"" completion:^(BOOL isSuccess, RongRTCCode desc) {
if (isSuccess) {
NSLog(@"发布自定义流成功");
}
}];
}
自定义一个 RongRTCAVOutputStream 流即可，使用此流发送屏幕共享数据。
2.2 开始发送屏幕共享数据
上面我们已经连接了融云的 IM 和加入了 RTC 房间，并且自定义了一个发送屏幕共享的自定义流，接下来，如何将此流发布出去呢？
- (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer withType:(RPSampleBufferType)sampleBufferType {
switch (sampleBufferType) {
case RPSampleBufferTypeVideo:
// Handle video sample buffer
[self.videoOutputStream write:sampleBuffer error:nil];
break;
case RPSampleBufferTypeAudioApp:
// Handle audio sample buffer for app audio
break;
case RPSampleBufferTypeAudioMic:
// Handle audio sample buffer for mic audio
break;
default:
break;
}
}
但我们接收到了苹果上报的数据之后，调用 RongRTCAVOutputStream 中的 write:error: 方法，将 sampleBuffer 发送给远端，至此，屏幕共享数据就发送出去啦。
[self.videoOutputStream write:sampleBuffer error:nil];
融云的核心代码就是通过上面的连接 IM，加入房间，发布自定义流，然后通过自定义流的 write:error: 方法将 sampleBuffer 发送出去。
不管是通过 ReplayKit 取得屏幕视频，还是使用 Socket 在进程间传输，都是为最终的 write:error: 服务。
总结
Extension内存是有限制的，最大 50M，所以在Extension里面处理数据需要格外注意内存释放；- 如果
VideotoolBox在后台解码一直失败，只需把VideotoolBox重启一下即可，此步骤在上面的代码中有体现； - 如果不需要将
Extension的数据传到主 App，只需在Extension里直接将流通过RongRTCLib发布出去即可，缺点是Extension中发布自定义流的用户与主 App 中的用户不是同一个，这也是上面通过Socket将数据传递给主App要解决的问题； - 如果主 App 需要拿到屏幕共享的数据处理，使用
Socket将流先发给主 App，然后在主 App 里面通过RongRTCLib将流发出去。
最后附上 Demo