感谢 STF 的开源，让 Android 设备远程控制变得简单，STF 通过 minicap 和 minitouch 实现设备的显示和控制。
STF 在实际使用中会发现一些棘手的问题
1.电视不支持 minitouch
2.新手机比如 mi8 mi9 不支持 minicap
3.Android 发布新版本需要适配 minicap
分享一个新的方法，来弥补这些不足
演示效果如下，由于图片较大（14MB），手机党请谨慎点击如下链接，建议完全加载完在播放，否则会卡，电脑的录屏软件很不给力，渣渣画质。。。
https://github.com/wenxiaomao1023/scrcpy/blob/master/assets/out.gif
app 目录 运行在 PC 端，对于 web 远程控制，这部分是不需要的
server 目录 运行在手机端，提供屏幕数据，接收并响应控制事件
1.获取 frame 数据方式是一致的（sdk19 以上）
2.Scrcpy 将 frame 编码 h264
3.minicap 将 frame 编码 jpeg
Scrcpy 处理方式看起来会更好，但是有一个问题，他的设计是将屏幕数据直接发给 PC，然后在 PC 上解码显示，这种方式在网页上却很不好展示。
1.Broadway 在前端解码 h264 并显示
2.wfs.js 在前端将 h264 转成 mp4 送给 h5 MSE 实现播放，这种类似直播，B 站 flv.js 那种
以上两种尝试都获得了图像，但个人感觉，以上两个方案感觉都有坑，还需要大量优化才能脱坑
当前摸索出的解决方法，Scrcpy 将 frame 编码 jpeg 发给前端然后通过画布展示，浏览器兼容好，可行性高，minicap 也是这么做的，修改方式见如下
https://github.com/wenxiaomao1023/scrcpy/commit/46d1c009d8ce559dc1ac1cdfceb234f1c7728498
1.使用 ImageReader 获取 frame 数据，通过 libjpeg-turbo 编码 jpeg
2.控制帧率，压缩率，缩放比例，可以减少带宽占用，提高流畅性
3.考虑到当前大多是 minicap 的方案，所以 scrcpy 返回的屏幕数据格式兼容了 minicap 的数据格式（banner+jpegsize+jpegdata），移植改动会很小
4.最低支持到 Android4.4（adb forward 命令有变化，见下文，默认端口 6612）
5.返回旋转状态（为了替换 STFService.apk）
6.添加获取 DumpHierarchy 信息（启动命令加-D 参数），获取界面布局信息为录制 Case 功能准备
1.德芙般丝滑，手机播放视频一点不卡，web 端展示也很流畅（30 - 50 FPS）
2.支持电视 touch
3.支持 mi8，mi9 等图像展示，不必在适配 minicap.so 啦，耶！
1.最低支持 android5.0，由于还依赖 android.system.Os，若想兼容低版本设备需要配合 minicap 使用，当前最低可支持到 Android4.4
2.流量问题，如果你的网站部署在内网，建议参考此方案，如果部署在公网，建议采用原生的 Scrcpy 方式
我已经编好了 ARMv7 (32-bit) 和 ARMv8 (64-bit)
https://github.com/wenxiaomao1023/scrcpy/tree/master/server/libs/libturbojpeg/prebuilt
如果你需要其他平台，可参考此文档 Building libjpeg-turbo for Android 部分
https://github.com/libjpeg-turbo/libjpeg-turbo/blob/master/BUILDING.md
如果不需要，可跳过此步骤
android sdk 里有 ninja，如 Android/Sdk/cmake/3.6.4111459/bin/ninja，加到环境变量里即可，meson 需要安装
如果不想安装这些，可以往下看，用 gradle 编译
git clone https://github.com/wenxiaomao1023/scrcpy.git
cd scrcpy
meson x --buildtype release --strip -Db_lto=true
ninja -Cx
编译后会在 scrcpy 目录下生成
x/server/scrcpy-server.jar
server/jniLibs/armeabi-v7a/libcompress.so
server/jniLibs/arm64-v8a/libcompress.so
git clone https://github.com/wenxiaomao1023/scrcpy.git
cd scrcpy/server
../gradlew assembleDebug
编译后会在 scrcpy 目录下生成
server/build/outputs/apk/debug/server-debug.apk
server/jniLibs/armeabi-v7a/libcompress.so
server/jniLibs/arm64-v8a/libcompress.so
server/build/outputs/apk/debug/server-debug.apk 和 x/server/scrcpy-server.jar 是一样的，下文中都按 scrcpy-server.jar 命名方式进行说明
# 先看下设备的abi，
adb shell getprop ro.product.cpu.abi
新版本添加-L 参数，LD_LIBRARY_PATH 的值等于-L 参数的返回值，并追加:/data/local/tmp
# armeabi-v7a
adb push scrcpy/server/jniLibs/armeabi-v7a/libcompress.so /data/local/tmp/
adb push scrcpy/server/libs/libturbojpeg/prebuilt/armeabi-v7a/libturbojpeg.so /data/local/tmp/
adb push scrcpy/x/server/scrcpy-server.jar /data/local/tmp/
adb shell chmod 777 /data/local/tmp/scrcpy-server.jar
adb shell CLASSPATH=/data/local/tmp/scrcpy-server.jar app_process / com.genymobile.scrcpy.Server -L
# LD_LIBRARY_PATH的值为上步-L的返回值加:/data/local/tmp（注意有个英文冒号）
adb shell LD_LIBRARY_PATH=???:/data/local/tmp CLASSPATH=/data/local/tmp/scrcpy-server.jar app_process / com.genymobile.scrcpy.Server
# arm64-v8a
adb push server/jniLibs/arm64-v8a/libcompress.so /data/local/tmp/
adb push server/libs/libturbojpeg/prebuilt/arm64-v8a/libturbojpeg.so /data/local/tmp/
adb push scrcpy/x/server/scrcpy-server.jar /data/local/tmp/
adb shell chmod 777 /data/local/tmp/scrcpy-server.jar
adb shell CLASSPATH=/data/local/tmp/scrcpy-server.jar app_process / com.genymobile.scrcpy.Server -L
# LD_LIBRARY_PATH的值为上步-L的返回值加:/data/local/tmp（注意有个英文冒号）
adb shell LD_LIBRARY_PATH=???:/data/local/tmp CLASSPATH=/data/local/tmp/scrcpy-server.jar app_process / com.genymobile.scrcpy.Server
# Android4.4设备
# 前几步同上（查看abi push libcompress.so libturbojpeg.so，push scrcpy-server.jar，chmod scrcpy-server.jar）
# 不同的地方在于需要指定ANDROID_DATA参数，否则启动会报错Dex cache directory isn't writable: /data/dalvik-cache (Permission denied) uid=2000 gid=2000
# 见解决方法https://stackoverflow.com/questions/21757935/running-android-java-based-command-line-utility-from-adb-shell
adb shell mkdir -p /data/local/tmp/dalvik-cache
adb shell ANDROID_DATA=/data/local/tmp CLASSPATH=/data/local/tmp/scrcpy-server.jar app_process / com.genymobile.scrcpy.Server -L
# LD_LIBRARY_PATH的值为上步-L的返回值加:/data/local/tmp（注意有个英文冒号）
adb shell ANDROID_DATA=/data/local/tmp LD_LIBRARY_PATH=???:/data/local/tmp CLASSPATH=/data/local/tmp/scrcpy-server.jar app_process / com.genymobile.scrcpy.Server
app_process / com.genymobile.scrcpy.Server 这个命令可以设置如下参数，建议使用命令如下，压缩质量 60，最高 24 帧，480P
app_process / com.genymobile.scrcpy.Server -Q 60 -r 24 -P 480
Usage: [-h]
jpeg:
-r <value>: Frame rate (frames/sec).
-P <value>: Display projection (1080, 720, 480, 360...).
-Q <value>: JPEG quality (0-100).
-c: Control only.
-L: Library path.
-D: Dump window hierarchy.
-h: Show help.
scrcpy-server.jar 兼容了 minicap 数据格式，可以直接用 minicap 的 demo app.js 看效果
https://github.com/openstf/minicap/tree/master/example
https://github.com/openstf/minicap/blob/master/example/app.js
需要把 app.js 改一下，多一个连接，修改如下
// 原始代码默认的图像socket
var stream = net.connect({
port: 1717
})
// 修改1 加一个控制socket，在默认的net.connect后再net.connect一次（必须）
var controlStream = net.connect({
port: 1717
})
// 修改2 新版本除了返回图像，还会返回旋转和层级（如果-D），需要容错处理，屏蔽掉exit()，并加else
if (frameBody[0] !== 0xFF || frameBody[1] !== 0xD8) {
console.error('Frame body does not start with JPG header', frameBody)
// process.exit(1) //屏蔽exit()
}
else { // 添加else
ws.send(frameBody, {
binary: true
})
}
git clone https://github.com/openstf/minicap.git
cd minicap/example
npm install
# 注意这里要改为localabstract:scrcpy（旧版本）
# 注意这里要改为tcp:6612（新版本）
adb forward tcp:1717 tcp:6612
node app.js
访问 http://127.0.0.1:9002
如果有时刷新会没效果，只显示一个红框，先确认是否执行了 adb forward tcp:1717 tcp:6612，如果是概率性刷新不出图像这是正常的，客户端连接失败需要做重连操作，例子里并没有，所以简单解决办法是重启下 scrcpy-server.jar，然后刷新网页
Scrcpy touch 的实现可以参考如下实现，当前实现常用的三种事件消息
// 键值 HOME，BACK，MENU 等
CONTROL_MSG_TYPE_INJECT_KEYCODE
// 点击和滑动
CONTROL_MSG_TYPE_INJECT_TOUCH_EVENT
// 鼠标滚轮滚动
CONTROL_MSG_TYPE_INJECT_SCROLL_EVENT
package main
import (
"errors"
"net"
"github.com/qiniu/log"
"bytes"
"encoding/binary"
)
type MessageType int8
const (
CONTROL_MSG_TYPE_INJECT_KEYCODE MessageType = iota
CONTROL_MSG_TYPE_INJECT_TEXT
CONTROL_MSG_TYPE_INJECT_TOUCH_EVENT
CONTROL_MSG_TYPE_INJECT_SCROLL_EVENT
CONTROL_MSG_TYPE_BACK_OR_SCREEN_ON
CONTROL_MSG_TYPE_EXPAND_NOTIFICATION_PANEL
CONTROL_MSG_TYPE_COLLAPSE_NOTIFICATION_PANEL
CONTROL_MSG_TYPE_GET_CLIPBOARD
CONTROL_MSG_TYPE_SET_CLIPBOARD
CONTROL_MSG_TYPE_SET_SCREEN_POWER_MODE
)
type PositionType struct {
X int32 `json:"x"`
Y int32 `json:"y"`
Width int16 `json:"width"`
Height int16 `json:"height"`
}
type Message struct {
Msg_type MessageType `json:"msg_type"`
// CONTROL_MSG_TYPE_INJECT_KEYCODE
Msg_inject_keycode_action int8 `json:"msg_inject_keycode_action"`
Msg_inject_keycode_keycode int32 `json:"msg_inject_keycode_keycode"`
Msg_inject_keycode_metastate int32 `json:"msg_inject_keycode_metastate"`
// CONTROL_MSG_TYPE_INJECT_TOUCH_EVENT
Msg_inject_touch_action int8 `json:"msg_inject_touch_action"`
Msg_inject_touch_pointerid int64 `json:"msg_inject_touch_pointerid"`
Msg_inject_touch_position PositionType `json:"msg_inject_touch_position"`
Msg_inject_touch_pressure uint16 `json:"msg_inject_touch_pressure"`
Msg_inject_touch_buttons int32 `json:"msg_inject_touch_buttons"`
// CONTROL_MSG_TYPE_INJECT_SCROLL_EVENT
Msg_inject_scroll_position PositionType `json:"msg_inject_scroll_position"`
Msg_inject_scroll_horizontal int32 `json:"msg_inject_scroll_horizontal"`
Msg_inject_scroll_vertical int32 `json:"msg_inject_scroll_vertical"`
}
type KeycodeMessage struct {
Msg_type MessageType `json:"msg_type"`
Msg_inject_keycode_action int8 `json:"msg_inject_keycode_action"`
Msg_inject_keycode_keycode int32 `json:"msg_inject_keycode_keycode"`
Msg_inject_keycode_metastate int32 `json:"msg_inject_keycode_metastate"`
}
type TouchMessage struct {
Msg_type MessageType `json:"msg_type"`
Msg_inject_touch_action int8 `json:"msg_inject_touch_action"`
Msg_inject_touch_pointerid int64 `json:"msg_inject_touch_pointerid"`
Msg_inject_touch_position PositionType `json:"msg_inject_touch_position"`
Msg_inject_touch_pressure uint16 `json:"msg_inject_touch_pressure"`
Msg_inject_touch_buttons int32 `json:"msg_inject_touch_buttons"`
}
type ScrollMessage struct {
Msg_type MessageType `json:"msg_type"`
Msg_inject_scroll_position PositionType `json:"msg_inject_scroll_position"`
Msg_inject_scroll_horizontal int32 `json:"msg_inject_scroll_horizontal"`
Msg_inject_scroll_vertical int32 `json:"msg_inject_scroll_vertical"`
}
func drainScrcpyRequests(conn net.Conn, reqC chan Message) error {
for req := range reqC {
var err error
switch req.Msg_type {
case CONTROL_MSG_TYPE_INJECT_KEYCODE:
t := KeycodeMessage{
Msg_type: req.Msg_type,
Msg_inject_keycode_action: req.Msg_inject_keycode_action,
Msg_inject_keycode_keycode: req.Msg_inject_keycode_keycode,
Msg_inject_keycode_metastate: req.Msg_inject_keycode_metastate,
}
buf := &bytes.Buffer{}
err := binary.Write(buf, binary.BigEndian, t)
if err != nil {
log.Debugf("CONTROL_MSG_TYPE_INJECT_KEYCODE error: %s", err)
log.Debugf("%s",buf.Bytes())
break
}
_, err = conn.Write(buf.Bytes())
case CONTROL_MSG_TYPE_INJECT_TEXT:
case CONTROL_MSG_TYPE_INJECT_TOUCH_EVENT:
var pointerid int64 = -1
var pressure uint16 = 65535
var buttons int32 = 1
req.Msg_inject_touch_pointerid = pointerid
req.Msg_inject_touch_pressure = pressure
req.Msg_inject_touch_buttons = buttons
t := TouchMessage{
Msg_type: req.Msg_type,
Msg_inject_touch_action: req.Msg_inject_touch_action,
Msg_inject_touch_pointerid: req.Msg_inject_touch_pointerid,
Msg_inject_touch_position: PositionType{
X: req.Msg_inject_touch_position.X,
Y: req.Msg_inject_touch_position.Y,
Width: req.Msg_inject_touch_position.Width,
Height: req.Msg_inject_touch_position.Height,
},
Msg_inject_touch_pressure: req.Msg_inject_touch_pressure,
Msg_inject_touch_buttons: req.Msg_inject_touch_buttons,
}
buf := &bytes.Buffer{}
err := binary.Write(buf, binary.BigEndian, t)
if err != nil {
log.Debugf("CONTROL_MSG_TYPE_INJECT_TOUCH_EVENT error: %s", err)
log.Debugf("%s",buf.Bytes())
break
}
_, err = conn.Write(buf.Bytes())
case CONTROL_MSG_TYPE_INJECT_SCROLL_EVENT:
t := ScrollMessage{
Msg_type: req.Msg_type,
Msg_inject_scroll_position: PositionType{
X: req.Msg_inject_scroll_position.X,
Y: req.Msg_inject_scroll_position.Y,
Width: req.Msg_inject_scroll_position.Width,
Height: req.Msg_inject_scroll_position.Height,
},
Msg_inject_scroll_horizontal: req.Msg_inject_scroll_horizontal,
Msg_inject_scroll_vertical: req.Msg_inject_scroll_vertical,
}
buf := &bytes.Buffer{}
err := binary.Write(buf, binary.BigEndian, t)
if err != nil {
log.Debugf("CONTROL_MSG_TYPE_INJECT_SCROLL_EVENT error: %s", err)
log.Debugf("%s",buf.Bytes())
break
}
_, err = conn.Write(buf.Bytes())
case CONTROL_MSG_TYPE_BACK_OR_SCREEN_ON:
case CONTROL_MSG_TYPE_EXPAND_NOTIFICATION_PANEL:
case CONTROL_MSG_TYPE_COLLAPSE_NOTIFICATION_PANEL:
case CONTROL_MSG_TYPE_GET_CLIPBOARD:
case CONTROL_MSG_TYPE_SET_CLIPBOARD:
case CONTROL_MSG_TYPE_SET_SCREEN_POWER_MODE:
default:
err = errors.New("unsupported msg type")
}
if err != nil {
return err
}
}
return nil
}
let scrcpyKey = (key) => {
ws.send(JSON.stringify({
"msg_type": 0,
"msg_inject_keycode_action": 0,
"msg_inject_keycode_keycode": key,
"msg_inject_keycode_metastate": 0
}))
ws.send(JSON.stringify({
"msg_type": 0,
"msg_inject_keycode_action": 1,
"msg_inject_keycode_keycode": key,
"msg_inject_keycode_metastate": 0
}))
}
let scrcpyTouchDown = (touch) => {
ws.send(JSON.stringify({
"msg_type": 2,
"msg_inject_touch_action": 0,
"msg_inject_touch_position": {
"x": touch.x, "y": touch.y, "width": touch.w, "height": touch.h
}}));
}
let scrcpyTouchMove = (touch) => {
ws.send(JSON.stringify({
"msg_type": 2,
"msg_inject_touch_action": 2,
"msg_inject_touch_position": {
"x": touch.x, "y": touch.y, "width": touch.w, "height": touch.h
}
}));
}
let scrcpyTouchUp = (touch) => {
ws.send(JSON.stringify({
"msg_type": 2,
"msg_inject_touch_action": 1,
"msg_inject_touch_position": {
"x": touch.x, "y": touch.y, "width": touch.w, "height": touch.h
}
}));
}
//向下滚动
let scrcpyScrollDown = (touch) => {
ws.send(JSON.stringify({
"msg_type": 3,
"msg_inject_scroll_position": {
"x": touch.x, "y": touch.y, "width": touch.w, "height": touch.h
},
"msg_inject_scroll_horizontal": 0,
"msg_inject_scroll_vertical": -1,
}));
}
//向上滚动
let scrcpyScrollUp = (touch) => {
ws.send(JSON.stringify({
"msg_type": 3,
"msg_inject_scroll_position": {
"x": touch.x, "y": touch.y, "width": touch.w, "height": touch.h
},
"msg_inject_scroll_horizontal": 0,
"msg_inject_scroll_vertical": 1,
}));
}
项目还在开发阶段，欢迎反馈问题 : )
收工～