Macaca [求助] macaca reliable-master 以 docker 方式部署完毕后,访问容器宿主机地址访问不通~
背景:
在非 docker 模式部署 reliable 问题迟迟得不到解决,参见前贴macaca reliable-master 创建分发任务报异常~,寻思活人不能在一棵树上吊死
,怎么才能"抱得美人 (reliable) 归",还得回归本源,docker 方式部署 reliable 再趟趟路 (没有一开始用 docker 方式部署,是因为现有服务器系统版本偏低),So 开始精心准备服务器等资源,一番"梳妆打扮"之后准备和 reliable 来一个第二次约会~~~
配置:
- Reliable master server OS->CentOS Linux release 7.5.1804 (Core)
- Reliable master server system 内核->3.10.0-862.14.4.el7.x86_64
- Reliable master server Docker 版本->Docker version 18.09.0, build 4d60db4
- Reliable slave OS->window7
过程:
Reliable Master 部署过程参考 --- 官方文档:Reliable Master 部署和文章Reliable Master 持续集成环境搭建 Centos
问题:
docker 方式部署 master 后,容器显示分配的是 IPv6 地址,外网访问不到容器相关应用!
备注:
1. reliable-docker-base 部署日志片段:
Step 15/20 : ENV NVM_NODEJS_ORG_MIRROR=https://npm.taobao.org/mirrors/node
---> Running in 82c8c96aab70
Removing intermediate container 82c8c96aab70
---> 782775ce2f90
Step 16/20 : RUN source $HOME/.nvm/nvm.sh && NVM_NODEJS_ORG_MIRROR=$NVM_NODEJS_ORG_MIRROR nvm install $NODE_VERSION
---> Running in 9aaf63d3a7c1
Downloading and installing node v4.5.0...
Downloading https://npm.taobao.org/mirrors/node/v4.5.0/node-v4.5.0-linux-x64.tar.xz...
######################################################################## 100.0%
Computing checksum with sha256sum
Checksums matched!
Now using node v4.5.0 (npm v2.15.9)
Creating default alias: default -> v4.5.0 *
Removing intermediate container 9aaf63d3a7c1
---> f0e706313743
Step 17/20 : ENV PATH="$ROOT_DIR/.nvm/versions/node/$NODE_VERSION/bin:$PATH"
---> Running in 916303b2c81c
Removing intermediate container 916303b2c81c
---> 500a23aecc1a
Step 18/20 : WORKDIR /
---> Running in 9ad3545f26ae
Removing intermediate container 9ad3545f26ae
---> 16837586a7d0
Step 19/20 : COPY ./entrypoint.sh /
---> 47351b143b6e
Step 20/20 : ENTRYPOINT ["/entrypoint.sh"]
---> Running in b4691109c898
Removing intermediate container b4691109c898
---> 70eb77d25385
Successfully built 70eb77d25385
2. reliable-master 部署日志片段:
istanbul@0.4.5 node_modules/istanbul
├── abbrev@1.0.9
├── async@1.5.2
├── wordwrap@1.0.0
├── nopt@3.0.6
├── esprima@2.7.3
├── once@1.4.0 (wrappy@1.0.2)
├── supports-color@3.2.3 (has-flag@1.0.0)
├── which@1.3.1 (isexe@2.0.0)
├── mkdirp@0.5.1 (minimist@0.0.8)
├── glob@5.0.15 (path-is-absolute@1.0.1, inherits@2.0.3, inflight@1.0.6, minimatch@3.0.4)
├── resolve@1.1.7
├── escodegen@1.8.1 (estraverse@1.9.3, esutils@2.0.2, optionator@0.8.2, source-map@0.2.0)
├── js-yaml@3.12.0 (esprima@4.0.1, argparse@1.0.10)
└── handlebars@4.0.12 (source-map@0.6.1, uglify-js@3.4.9, optimist@0.6.1, async@2.6.1)
Hash: 7b9f2227ac0913b257fe
Version: webpack 1.15.0
Time: 3534ms
Asset Size Chunks Chunk Names
auth.js 3.78 kB 0 [emitted] auth
dashboard.js 1.14 MB 1 [emitted] dashboard
history.js 1.14 MB 2 [emitted] history
home.js 1.14 MB 3 [emitted] home
index.js 2.42 kB 4 [emitted] index
login.js 3.34 kB 5 [emitted] login
password.js 4.49 kB 6 [emitted] password
signup.js 3.99 kB 7 [emitted] signup
task.js 1.14 MB 8 [emitted] task
user.js 1.4 kB 9 [emitted] user
+ 191 hidden modules
Removing intermediate container bd0772b67604
---> bb220ef70975
Successfully built bb220ef70975
Successfully tagged reliable-master:latest
3.docker-compose 启动日志片段:
RELIABLE_MASTER_PORT=8080 docker-compose down
RELIABLE_MASTER_PORT=8080 docker-compose build
redis uses an image, skipping
mongo uses an image, skipping
reliable-master uses an image, skipping
RELIABLE_MASTER_PORT=8080 docker-compose up -d --remove-orphans
Pulling redis (redis:)...
latest: Pulling from library/redis
f17d81b4b692: Pull complete
b32474098757: Pull complete
8980cabe8bc2: Pull complete
e614c66c2b9c: Pull complete
6eb43ec9256b: Pull complete
394ecf5f46d4: Pull complete
Pulling mongo (mongo:)...
latest: Pulling from library/mongo
18d680d61657: Pull complete
0addb6fece63: Pull complete
78e58219b215: Pull complete
eb6959a66df2: Pull complete
1bb66a4db707: Pull complete
b18fa018e44b: Pull complete
5d0142bcb0df: Pull complete
db65733de31a: Pull complete
867fe12df2c5: Pull complete
29aa76034f8f: Pull complete
28e61706e03f: Pull complete
98ec6a7c0d50: Pull complete
c752faaa2e63: Pull complete
Creating reliable_mongo ... done
Creating reliable_redis ... done
Creating reliable-master ... done
4.docker images:
REPOSITORY TAG IMAGE ID CREATED SIZE
reliable-master latest bb220ef70975 7 minutes ago 1.4GB
reliable-docker-base latest 70eb77d25385 26 minutes ago 1.21GB
redis latest 415381a6cb81 5 days ago 94.9MB
mongo latest ea6441073322 5 days ago 382MB
centos 7 75835a67d134 5 weeks ago 200MB
hello-world latest 4ab4c602aa5e 2 months ago 1.84kB
6.docker ps:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
391eb97fe307 reliable-master "/entrypoint.sh /rel…" 2 minutes ago Up 2 minutes 0.0.0.0:8080->8080/tcp reliable-master###宿主机8080端口已经映射至容器端口
6f55379e7197 mongo "docker-entrypoint.s…" 2 minutes ago Up 2 minutes 27017/tcp reliable_mongo
c26f1dc6b35c redis "docker-entrypoint.s…" 2 minutes ago Up 2 minutes 6379/tcp reliable_redis
5.查看端口是否启动 sudo lsof -OnP | grep LISTEN:
sshd 570 root 3u IPv4 18949 0t0 TCP *:22 (LISTEN)
sshd 570 root 4u IPv6 18951 0t0 TCP *:22 (LISTEN)
master 833 root 13u IPv4 15726 0t0 TCP 127.0.0.1:25 (LISTEN)
master 833 root 14u IPv6 15727 0t0 TCP [::1]:25 (LISTEN)
docker-pr 15111 root 4u IPv6 333445 0t0 TCP *:8080 (LISTEN)
docker-pr 15111 15112 root 4u IPv6 333445 0t0 TCP *:8080 (LISTEN)
docker-pr 15111 15113 root 4u IPv6 333445 0t0 TCP *:8080 (LISTEN)
docker-pr 15111 15114 root 4u IPv6 333445 0t0 TCP *:8080 (LISTEN)
docker-pr 15111 15115 root 4u IPv6 333445 0t0 TCP *:8080 (LISTEN)
docker-pr 15111 15116 root 4u IPv6 333445 0t0 TCP *:8080 (LISTEN)
docker-pr 15111 15117 root 4u IPv6 333445 0t0 TCP *:8080 (LISTEN)
docker-pr 15111 15118 root 4u IPv6 333445 0t0 TCP *:8080 (LISTEN)
docker-pr 15111 15119 root 4u IPv6 333445 0t0 TCP *:8080 (LISTEN)
dockerd 22611 root 5u IPv6 226200 0t0 TCP *:2375 (LISTEN)
dockerd 22611 root 7u IPv6 226203 0t0 TCP *:7654 (LISTEN)
dockerd 22611 22612 root 5u IPv6 226200 0t0 TCP *:2375 (LISTEN)
dockerd 22611 22612 root 7u IPv6 226203 0t0 TCP *:7654 (LISTEN)
dockerd 22611 22613 root 5u IPv6 226200 0t0 TCP *:2375 (LISTEN)
dockerd 22611 22613 root 7u IPv6 226203 0t0 TCP *:7654 (LISTEN)
dockerd 22611 22614 root 5u IPv6 226200 0t0 TCP *:2375 (LISTEN)
dockerd 22611 22614 root 7u IPv6 226203 0t0 TCP *:7654 (LISTEN)
dockerd 22611 22615 root 5u IPv6 226200 0t0 TCP *:2375 (LISTEN)
dockerd 22611 22615 root 7u IPv6 226203 0t0 TCP *:7654 (LISTEN)
dockerd 22611 22616 root 5u IPv6 226200 0t0 TCP *:2375 (LISTEN)
dockerd 22611 22616 root 7u IPv6 226203 0t0 TCP *:7654 (LISTEN)
dockerd 22611 22617 root 5u IPv6 226200 0t0 TCP *:2375 (LISTEN)
dockerd 22611 22617 root 7u IPv6 226203 0t0 TCP *:7654 (LISTEN)
dockerd 22611 22618 root 5u IPv6 226200 0t0 TCP *:2375 (LISTEN)
dockerd 22611 22618 root 7u IPv6 226203 0t0 TCP *:7654 (LISTEN)
dockerd 22611 22619 root 5u IPv6 226200 0t0 TCP *:2375 (LISTEN)
dockerd 22611 22619 root 7u IPv6 226203 0t0 TCP *:7654 (LISTEN)
dockerd 22611 22620 root 5u IPv6 226200 0t0 TCP *:2375 (LISTEN)
dockerd 22611 22620 root 7u IPv6 226203 0t0 TCP *:7654 (LISTEN)
dockerd 22611 22621 root 5u IPv6 226200 0t0 TCP *:2375 (LISTEN)
dockerd 22611 22621 root 7u IPv6 226203 0t0 TCP *:7654 (LISTEN)
dockerd 22611 22622 root 5u IPv6 226200 0t0 TCP *:2375 (LISTEN)
dockerd 22611 22622 root 7u IPv6 226203 0t0 TCP *:7654 (LISTEN)
dockerd 22611 22624 root 5u IPv6 226200 0t0 TCP *:2375 (LISTEN)
dockerd 22611 22624 root 7u IPv6 226203 0t0 TCP *:7654 (LISTEN)
dockerd 22611 22625 root 5u IPv6 226200 0t0 TCP *:2375 (LISTEN)
dockerd 22611 22625 root 7u IPv6 226203 0t0 TCP *:7654 (LISTEN)
dockerd 22611 22626 root 5u IPv6 226200 0t0 TCP *:2375 (LISTEN)
dockerd 22611 22626 root 7u IPv6 226203 0t0 TCP *:7654 (LISTEN)
dockerd 22611 22627 root 5u IPv6 226200 0t0 TCP *:2375 (LISTEN)
dockerd 22611 22627 root 7u IPv6 226203 0t0 TCP *:7654 (LISTEN)
dockerd 22611 22875 root 5u IPv6 226200 0t0 TCP *:2375 (LISTEN)
dockerd 22611 22875 root 7u IPv6 226203 0t0 TCP *:7654 (LISTEN)
dockerd 22611 23101 root 5u IPv6 226200 0t0 TCP *:2375 (LISTEN)
dockerd 22611 23101 root 7u IPv6 226203 0t0 TCP *:7654 (LISTEN)
dockerd 22611 25032 root 5u IPv6 226200 0t0 TCP *:2375 (LISTEN)
dockerd 22611 25032 root 7u IPv6 226203 0t0 TCP *:7654 (LISTEN)
dockerd 22611 25075 root 5u IPv6 226200 0t0 TCP *:2375 (LISTEN)
dockerd 22611 25075 root 7u IPv6 226203 0t0 TCP *:7654 (LISTEN)
6.通过外网访问容器宿主机,显示连接不通:
7.查看防火墙状态,应该也是关闭的-systemctl status firewalld :
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
8.查看监听服务的状态-netstat -tulpn :
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp6 0 0 ::1:25 :::* LISTEN -
tcp6 0 0 :::7654 :::* LISTEN -
tcp6 0 0 :::2375 :::* LISTEN -
tcp6 0 0 :::8080 :::* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -
udp 0 0 0.0.0.0:68 0.0.0.0:* -
9.查看 Reliable 服务器进程:
[xxxxadmin@RZ-SI-TEST-01 reliable-macaca-docker-compose]$ ps aux |grep docker
root 22611 0.4 0.4 1005204 69980 ? Ssl 10:48 0:51 /usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock -H tcp://0.0.0.0:7654
root 24019 0.0 0.0 10732 3808 ? Sl 12:54 0:00 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/2f6e49ab5317c855fff7ac90e8a28001415d3b84f3de275172f881d50ba83456 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
root 24033 0.0 0.0 10732 3904 ? Sl 12:54 0:00 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/664a60b164799c897a142a596bd256db4efa1b9c66268db007a47271262d305e -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
root 24173 0.0 0.0 264624 2636 ? Sl 12:54 0:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8080 -container-ip 172.17.0.4 -container-port 8080
root 24181 0.0 0.0 10732 3828 ? Sl 12:54 0:00 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/3730d02ae29747881af74da24af6645a6addd382795042567bd2e92878cf0a93 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
xxxxadm+ 31179 0.0 0.0 112708 936 pts/0 S+ 13:59 0:00 grep --color=auto docker
10.reliable-master 容器日志:
> >> index.js:36:12 [worker:5] pid:42 middleware: static registed
> >> router.js:41:10 [worker:5] pid:42 router set
> >> index.js:23:10 [worker:8] pid:60 workder init with config:
> { server:
> { worker: 8,
> port: '127.0.0.1:8888:8888',
> protocol: 'http',
> verbose: true },
> database: 'mongodb://172.17.0.3/reliable',
> site:
> { title: 'Reliable',
> baseurl: 'http://reliable-test.com',
> locale: 'en-US',
> docurl: 'reliablejs.github.io',
> issueurl: 'https://github.com/alibaba/reliable/issues',
> login: true },
> auth:
> { github:
> { client_id: '8bb3d4f7fa7d3d346a58',
> client_secret: '416bdc362cefb378587aa75c1db9bdd4c84a3461' },
> gitlab:
> { protocol: 'http',
> server_url: '127.0.0.1:3000',
> client_id: '8bb3d4f7fa7d3d346a58',
> client_secret: '416bdc362cefb378587aa75c1db9bdd4c84a3461' } },
> mail:
> { name: 'reliable test',
> port: 465,
> host: 'smtp.reliable-test.com',
> secure: true,
> ignoreTLS: true,
> auth: { user: 'test@reliable-test.com', pass: 'reliable' },
> sloganImage: 'https://avatars0.githubusercontent.com/u/9263042?v=3&s=200' },
> plugins: {},
> pluginTextPrefix: 'reliable-plugin',
> pkg:
> { name: 'reliable-master',
> version: '2.0.0',
> description: 'Distributed reliable continuous integration service.',
> keywords: [ 'ci', 'automation', [length]: 2 ],
> bin: { 'reliable-master': './bin/reliable-master' },
> repository:
> { type: 'git',
> url: 'git@github.com:reliablejs/reliable-master.git' },
> dependencies:
> { 'ansi-to-html': '~0.3.0',
> babel: '~5.6.14',
> badgeboard: '~0.1.1',
> bcryptjs: '~2.2.1',
> chalk: '~1.1.1',
> co: '~4.6',
> 'co-body': '~1.1.0',
> 'co-request': '^0.2.1',
> commander: '~2.0.0',
> 'detect-port': '~0.1.3',
> koa: '~0.21.0',
> 'koa-csrf': '2.1.3',
> 'koa-favicon': '~1.2.0',
> 'koa-generic-session': '~1.10.0',
> 'koa-redis': '~1.0.1',
> 'koa-router': '~3.7.0',
> 'koa-static': '~1.4.7',
> mongoose: '=4.1.5',
> nodemailer: '~1.4.0',
> 'npm-update': '1.0.2',
> react: '~0.14.0',
> 'react-dom': '~0.14.2',
> redis: '~2.3.1',
> 'reliable-events': '^0.1.4',
> 'reliable-github-oauth': '^1.0.0',
> 'reliable-gitlab-oauth': '^1.0.0',
> 'reliable-logger': '^1.0.7',
> 'reliable-mail': '^0.2.0',
> 'reliable-plugin': '^0.1.9',
> validator: '~3.37.0',
> xutil: '~1.0.0',
> yamljs: '~0.2.4',
> zmq: '2.15.3' },
> devDependencies:
> { 'co-mocha': '~1.1.2',
> 'command-line-test': '^1.0.5',
> eslint: '~1.1.0',
> 'eslint-plugin-react': '~3.2.3',
> 'git-contributor': '^1.0.8',
> istanbul: '*',
> 'json-loader': '~0.5.2',
> 'jsx-loader': '~0.13.2',
> killing: '~1.0.1',
> mocha: '~2.4.5',
> 'pre-commit': '~1.1.1',
> 'react-d3-components': '~0.6.0',
> should: '*',
> webpack: '^1.13.1' },
> scripts:
> { test: 'make test',
> lint: 'make lint',
> contributor: 'git-contributor' },
> 'pre-commit': [ 'lint', [length]: 1 ],
> engines: { node: '>=8.12.0' },
> homepage: 'https://reliablejs.github.io/reliable',
> license: 'MIT' } }
> >> index.js:32:10 [worker:4] pid:36 base middlewares attached
> >> index.js:36:12 [worker:4] pid:36 middleware: i18n registed
> >> render.js:15:8 [worker:4] pid:36 render view path: /reliable-master/web/views
> >> index.js:36:12 [worker:4] pid:36 middleware: inject registed
> >> index.js:67:14 [master] pid:19 listening worker id: 5, pid: 42, address: 172.17.0.4:-1
> >> index.js:32:10 [worker:8] pid:60 base middlewares attached
> >> index.js:36:12 [worker:8] pid:60 middleware: i18n registed
> >> render.js:15:8 [worker:8] pid:60 render view path: /reliable-master/web/views
> >> index.js:36:12 [worker:4] pid:36 middleware: favicon registed
> >> index.js:36:12 [worker:8] pid:60 middleware: inject registed
> >> index.js:36:12 [worker:4] pid:36 middleware: powerby registed
> >> index.js:36:12 [worker:8] pid:60 middleware: favicon registed
> >> index.js:36:12 [worker:4] pid:36 middleware: static registed
> >> index.js:36:12 [worker:8] pid:60 middleware: powerby registed
> >> router.js:41:10 [worker:4] pid:36 router set
> >> index.js:36:12 [worker:8] pid:60 middleware: static registed
> >> index.js:67:14 [master] pid:19 listening worker id: 4, pid: 36, address: 172.17.0.4:-1
> >> router.js:41:10 [worker:8] pid:60 router set
> >> index.js:67:14 [master] pid:19 listening worker id: 8, pid: 60, address: 172.17.0.4:-1
> >> Server start at 2018-11-15 17:51:26 | http://172.17.0.4:-1
> >> run.js:89:14 [master] pid:19 slaves all offline with data
> { message: 'project' }
> >> manager.js:144:16 [master] pid:19 no slave for monitor #1
> >> run.js:89:14 [master] pid:19 slaves all offline with data
> { message: 'project' }
> >> run.js:89:14 [master] pid:19 slaves all offline with data
> { message: 'project' }
> >> run.js:89:14 [master] pid:19 slaves all offline with data
> { message: 'dispatch' }
> >> manager.js:144:16 [master] pid:19 no slave for monitor #1
> >> run.js:89:14 [master] pid:19 slaves all offline with data
> { message: 'project' }
> >> manager.js:144:16 [master] pid:19 no slave for monitor #1
> >> run.js:89:14 [master] pid:19 slaves all offline with data
> { message: 'project' }
分析:
1.通过端口查看发现 docker-proxy 使用的协议是 ipv6 协议做的代理映射
2.尝试启用 IPv6 的分组转发,修改 /etc/sysctl.conf: net.ipv6.conf.all.forwarding=1,结果失败~
3.另外尝试服务器禁用 ipv6,重新构建后分配的还是 ipv6,结果失败~
4.最后通过自定义网桥的方式,试着解决该问题,结果还是失败~
写到最后:目前该问题还在探索中,正应了那句话:生命不息,探索不止! 也盼望 macaca 开发团队@xdf等人或踩过类似坑儿的人看到此贴,给予指点帮助,谢谢 !
1. 参考:Port redirecting binding to IPv6 but not IPv4 interfaces
2. 参考:centos7 上如何禁用 ipv6
3. 参考:Docker-网络配置
「原创声明：保留所有权利，禁止转载」
