Macaca [求助] macaca reliable-master 以 docker 方式部署完毕后,访问容器宿主机地址访问不通~

PEPSI · 2018年11月16日 · 最后由 Mr Wang 回复于 2019年02月14日 · 2635 次阅读

背景:

在非 docker 模式部署 reliable 问题迟迟得不到解决,参见前贴macaca reliable-master 创建分发任务报异常~,寻思活人不能在一棵树上吊死😄 ,怎么才能"抱得美人 (reliable) 归",还得回归本源,docker 方式部署 reliable 再趟趟路 (没有一开始用 docker 方式部署,是因为现有服务器系统版本偏低),So 开始精心准备服务器等资源,一番"梳妆打扮"之后准备和 reliable 来一个第二次约会~~~

配置:

- Reliable master server OS->CentOS Linux release 7.5.1804 (Core)
- Reliable master server system 内核->3.10.0-862.14.4.el7.x86_64
- Reliable master server Docker 版本->Docker version 18.09.0, build 4d60db4
- Reliable slave OS->window7

过程:

Reliable Master 部署过程参考 --- 官方文档:Reliable Master 部署和文章Reliable Master 持续集成环境搭建 Centos

问题:

docker 方式部署 master 后,容器显示分配的是 IPv6 地址,外网访问不到容器相关应用!

备注:

1. reliable-docker-base 部署日志片段:

Step 15/20 : ENV NVM_NODEJS_ORG_MIRROR=https://npm.taobao.org/mirrors/node
 ---> Running in 82c8c96aab70
Removing intermediate container 82c8c96aab70
 ---> 782775ce2f90
Step 16/20 : RUN source $HOME/.nvm/nvm.sh   && NVM_NODEJS_ORG_MIRROR=$NVM_NODEJS_ORG_MIRROR nvm install $NODE_VERSION
 ---> Running in 9aaf63d3a7c1
Downloading and installing node v4.5.0...
Downloading https://npm.taobao.org/mirrors/node/v4.5.0/node-v4.5.0-linux-x64.tar.xz...
######################################################################## 100.0%
Computing checksum with sha256sum
Checksums matched!
Now using node v4.5.0 (npm v2.15.9)
Creating default alias: default -> v4.5.0 *
Removing intermediate container 9aaf63d3a7c1
 ---> f0e706313743
Step 17/20 : ENV PATH="$ROOT_DIR/.nvm/versions/node/$NODE_VERSION/bin:$PATH"
 ---> Running in 916303b2c81c
Removing intermediate container 916303b2c81c
 ---> 500a23aecc1a
Step 18/20 : WORKDIR /
 ---> Running in 9ad3545f26ae
Removing intermediate container 9ad3545f26ae
 ---> 16837586a7d0
Step 19/20 : COPY ./entrypoint.sh /
 ---> 47351b143b6e
Step 20/20 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Running in b4691109c898
Removing intermediate container b4691109c898
 ---> 70eb77d25385
Successfully built 70eb77d25385

2. reliable-master 部署日志片段:

 istanbul@0.4.5 node_modules/istanbul
├── abbrev@1.0.9
├── async@1.5.2
├── wordwrap@1.0.0
├── nopt@3.0.6
├── esprima@2.7.3
├── once@1.4.0 (wrappy@1.0.2)
├── supports-color@3.2.3 (has-flag@1.0.0)
├── which@1.3.1 (isexe@2.0.0)
├── mkdirp@0.5.1 (minimist@0.0.8)
├── glob@5.0.15 (path-is-absolute@1.0.1, inherits@2.0.3, inflight@1.0.6, minimatch@3.0.4)
├── resolve@1.1.7
├── escodegen@1.8.1 (estraverse@1.9.3, esutils@2.0.2, optionator@0.8.2, source-map@0.2.0)
├── js-yaml@3.12.0 (esprima@4.0.1, argparse@1.0.10)
└── handlebars@4.0.12 (source-map@0.6.1, uglify-js@3.4.9, optimist@0.6.1, async@2.6.1)
Hash: 7b9f2227ac0913b257fe
Version: webpack 1.15.0
Time: 3534ms
       Asset     Size  Chunks             Chunk Names
     auth.js  3.78 kB       0  [emitted]  auth
dashboard.js  1.14 MB       1  [emitted]  dashboard
  history.js  1.14 MB       2  [emitted]  history
     home.js  1.14 MB       3  [emitted]  home
    index.js  2.42 kB       4  [emitted]  index
    login.js  3.34 kB       5  [emitted]  login
 password.js  4.49 kB       6  [emitted]  password
   signup.js  3.99 kB       7  [emitted]  signup
     task.js  1.14 MB       8  [emitted]  task
     user.js   1.4 kB       9  [emitted]  user
    + 191 hidden modules
Removing intermediate container bd0772b67604
 ---> bb220ef70975
Successfully built bb220ef70975
Successfully tagged reliable-master:latest

3.docker-compose 启动日志片段:

RELIABLE_MASTER_PORT=8080 docker-compose down
RELIABLE_MASTER_PORT=8080 docker-compose build
redis uses an image, skipping
mongo uses an image, skipping
reliable-master uses an image, skipping
RELIABLE_MASTER_PORT=8080 docker-compose up -d --remove-orphans
Pulling redis (redis:)...
latest: Pulling from library/redis
f17d81b4b692: Pull complete
b32474098757: Pull complete
8980cabe8bc2: Pull complete
e614c66c2b9c: Pull complete
6eb43ec9256b: Pull complete
394ecf5f46d4: Pull complete
Pulling mongo (mongo:)...
latest: Pulling from library/mongo
18d680d61657: Pull complete
0addb6fece63: Pull complete
78e58219b215: Pull complete
eb6959a66df2: Pull complete
1bb66a4db707: Pull complete
b18fa018e44b: Pull complete
5d0142bcb0df: Pull complete
db65733de31a: Pull complete
867fe12df2c5: Pull complete
29aa76034f8f: Pull complete
28e61706e03f: Pull complete
98ec6a7c0d50: Pull complete
c752faaa2e63: Pull complete
Creating reliable_mongo ... done
Creating reliable_redis ... done
Creating reliable-master ... done

4.docker images:

REPOSITORY             TAG                 IMAGE ID            CREATED             SIZE
reliable-master        latest              bb220ef70975        7 minutes ago       1.4GB
reliable-docker-base   latest              70eb77d25385        26 minutes ago      1.21GB
redis                  latest              415381a6cb81        5 days ago          94.9MB
mongo                  latest              ea6441073322        5 days ago          382MB
centos                 7                   75835a67d134        5 weeks ago         200MB
hello-world            latest              4ab4c602aa5e        2 months ago        1.84kB

6.docker ps:

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
391eb97fe307        reliable-master     "/entrypoint.sh /rel…"   2 minutes ago       Up 2 minutes        0.0.0.0:8080->8080/tcp   reliable-master###宿主机8080端口已经映射至容器端口
6f55379e7197        mongo               "docker-entrypoint.s…"   2 minutes ago       Up 2 minutes        27017/tcp                reliable_mongo
c26f1dc6b35c        redis               "docker-entrypoint.s…"   2 minutes ago       Up 2 minutes        6379/tcp                 reliable_redis

5.查看端口是否启动 sudo lsof -OnP | grep LISTEN:

sshd        570            root    3u     IPv4              18949       0t0        TCP *:22 (LISTEN)
sshd        570            root    4u     IPv6              18951       0t0        TCP *:22 (LISTEN)
master      833            root   13u     IPv4              15726       0t0        TCP 127.0.0.1:25 (LISTEN)
master      833            root   14u     IPv6              15727       0t0        TCP [::1]:25 (LISTEN)
docker-pr 15111            root    4u     IPv6             333445       0t0        TCP *:8080 (LISTEN)
docker-pr 15111 15112      root    4u     IPv6             333445       0t0        TCP *:8080 (LISTEN)
docker-pr 15111 15113      root    4u     IPv6             333445       0t0        TCP *:8080 (LISTEN)
docker-pr 15111 15114      root    4u     IPv6             333445       0t0        TCP *:8080 (LISTEN)
docker-pr 15111 15115      root    4u     IPv6             333445       0t0        TCP *:8080 (LISTEN)
docker-pr 15111 15116      root    4u     IPv6             333445       0t0        TCP *:8080 (LISTEN)
docker-pr 15111 15117      root    4u     IPv6             333445       0t0        TCP *:8080 (LISTEN)
docker-pr 15111 15118      root    4u     IPv6             333445       0t0        TCP *:8080 (LISTEN)
docker-pr 15111 15119      root    4u     IPv6             333445       0t0        TCP *:8080 (LISTEN)
dockerd   22611            root    5u     IPv6             226200       0t0        TCP *:2375 (LISTEN)
dockerd   22611            root    7u     IPv6             226203       0t0        TCP *:7654 (LISTEN)
dockerd   22611 22612      root    5u     IPv6             226200       0t0        TCP *:2375 (LISTEN)
dockerd   22611 22612      root    7u     IPv6             226203       0t0        TCP *:7654 (LISTEN)
dockerd   22611 22613      root    5u     IPv6             226200       0t0        TCP *:2375 (LISTEN)
dockerd   22611 22613      root    7u     IPv6             226203       0t0        TCP *:7654 (LISTEN)
dockerd   22611 22614      root    5u     IPv6             226200       0t0        TCP *:2375 (LISTEN)
dockerd   22611 22614      root    7u     IPv6             226203       0t0        TCP *:7654 (LISTEN)
dockerd   22611 22615      root    5u     IPv6             226200       0t0        TCP *:2375 (LISTEN)
dockerd   22611 22615      root    7u     IPv6             226203       0t0        TCP *:7654 (LISTEN)
dockerd   22611 22616      root    5u     IPv6             226200       0t0        TCP *:2375 (LISTEN)
dockerd   22611 22616      root    7u     IPv6             226203       0t0        TCP *:7654 (LISTEN)
dockerd   22611 22617      root    5u     IPv6             226200       0t0        TCP *:2375 (LISTEN)
dockerd   22611 22617      root    7u     IPv6             226203       0t0        TCP *:7654 (LISTEN)
dockerd   22611 22618      root    5u     IPv6             226200       0t0        TCP *:2375 (LISTEN)
dockerd   22611 22618      root    7u     IPv6             226203       0t0        TCP *:7654 (LISTEN)
dockerd   22611 22619      root    5u     IPv6             226200       0t0        TCP *:2375 (LISTEN)
dockerd   22611 22619      root    7u     IPv6             226203       0t0        TCP *:7654 (LISTEN)
dockerd   22611 22620      root    5u     IPv6             226200       0t0        TCP *:2375 (LISTEN)
dockerd   22611 22620      root    7u     IPv6             226203       0t0        TCP *:7654 (LISTEN)
dockerd   22611 22621      root    5u     IPv6             226200       0t0        TCP *:2375 (LISTEN)
dockerd   22611 22621      root    7u     IPv6             226203       0t0        TCP *:7654 (LISTEN)
dockerd   22611 22622      root    5u     IPv6             226200       0t0        TCP *:2375 (LISTEN)
dockerd   22611 22622      root    7u     IPv6             226203       0t0        TCP *:7654 (LISTEN)
dockerd   22611 22624      root    5u     IPv6             226200       0t0        TCP *:2375 (LISTEN)
dockerd   22611 22624      root    7u     IPv6             226203       0t0        TCP *:7654 (LISTEN)
dockerd   22611 22625      root    5u     IPv6             226200       0t0        TCP *:2375 (LISTEN)
dockerd   22611 22625      root    7u     IPv6             226203       0t0        TCP *:7654 (LISTEN)
dockerd   22611 22626      root    5u     IPv6             226200       0t0        TCP *:2375 (LISTEN)
dockerd   22611 22626      root    7u     IPv6             226203       0t0        TCP *:7654 (LISTEN)
dockerd   22611 22627      root    5u     IPv6             226200       0t0        TCP *:2375 (LISTEN)
dockerd   22611 22627      root    7u     IPv6             226203       0t0        TCP *:7654 (LISTEN)
dockerd   22611 22875      root    5u     IPv6             226200       0t0        TCP *:2375 (LISTEN)
dockerd   22611 22875      root    7u     IPv6             226203       0t0        TCP *:7654 (LISTEN)
dockerd   22611 23101      root    5u     IPv6             226200       0t0        TCP *:2375 (LISTEN)
dockerd   22611 23101      root    7u     IPv6             226203       0t0        TCP *:7654 (LISTEN)
dockerd   22611 25032      root    5u     IPv6             226200       0t0        TCP *:2375 (LISTEN)
dockerd   22611 25032      root    7u     IPv6             226203       0t0        TCP *:7654 (LISTEN)
dockerd   22611 25075      root    5u     IPv6             226200       0t0        TCP *:2375 (LISTEN)
dockerd   22611 25075      root    7u     IPv6             226203       0t0        TCP *:7654 (LISTEN)

6.通过外网访问容器宿主机,显示连接不通:

7.查看防火墙状态,应该也是关闭的-systemctl status firewalld :

● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

8.查看监听服务的状态-netstat -tulpn :

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -                   
tcp6       0      0 ::1:25                  :::*                    LISTEN      -                   
tcp6       0      0 :::7654                 :::*                    LISTEN      -                   
tcp6       0      0 :::2375                 :::*                    LISTEN      -                   
tcp6       0      0 :::8080                 :::*                    LISTEN      -                   
tcp6       0      0 :::22                   :::*                    LISTEN      -                   
udp        0      0 0.0.0.0:68              0.0.0.0:*                           -   

9.查看 Reliable 服务器进程:

[xxxxadmin@RZ-SI-TEST-01 reliable-macaca-docker-compose]$ ps aux |grep docker
root     22611  0.4  0.4 1005204 69980 ?       Ssl  10:48   0:51 /usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock -H tcp://0.0.0.0:7654
root     24019  0.0  0.0  10732  3808 ?        Sl   12:54   0:00 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/2f6e49ab5317c855fff7ac90e8a28001415d3b84f3de275172f881d50ba83456 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
root     24033  0.0  0.0  10732  3904 ?        Sl   12:54   0:00 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/664a60b164799c897a142a596bd256db4efa1b9c66268db007a47271262d305e -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
root     24173  0.0  0.0 264624  2636 ?        Sl   12:54   0:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8080 -container-ip 172.17.0.4 -container-port 8080
root     24181  0.0  0.0  10732  3828 ?        Sl   12:54   0:00 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/3730d02ae29747881af74da24af6645a6addd382795042567bd2e92878cf0a93 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
xxxxadm+ 31179  0.0  0.0 112708   936 pts/0    S+   13:59   0:00 grep --color=auto docker

10.reliable-master 容器日志:

> >> index.js:36:12 [worker:5] pid:42 middleware: static registed
> >> router.js:41:10 [worker:5] pid:42 router set
> >> index.js:23:10 [worker:8] pid:60 workder init with config:
>  { server: 
>    { worker: 8,
>      port: '127.0.0.1:8888:8888',
>      protocol: 'http',
>      verbose: true },
>   database: 'mongodb://172.17.0.3/reliable',
>   site: 
>    { title: 'Reliable',
>      baseurl: 'http://reliable-test.com',
>      locale: 'en-US',
>      docurl: 'reliablejs.github.io',
>      issueurl: 'https://github.com/alibaba/reliable/issues',
>      login: true },
>   auth: 
>    { github: 
>       { client_id: '8bb3d4f7fa7d3d346a58',
>         client_secret: '416bdc362cefb378587aa75c1db9bdd4c84a3461' },
>      gitlab: 
>       { protocol: 'http',
>         server_url: '127.0.0.1:3000',
>         client_id: '8bb3d4f7fa7d3d346a58',
>         client_secret: '416bdc362cefb378587aa75c1db9bdd4c84a3461' } },
>   mail: 
>    { name: 'reliable test',
>      port: 465,
>      host: 'smtp.reliable-test.com',
>      secure: true,
>      ignoreTLS: true,
>      auth: { user: 'test@reliable-test.com', pass: 'reliable' },
>      sloganImage: 'https://avatars0.githubusercontent.com/u/9263042?v=3&s=200' },
>   plugins: {},
>   pluginTextPrefix: 'reliable-plugin',
>   pkg: 
>    { name: 'reliable-master',
>      version: '2.0.0',
>      description: 'Distributed reliable continuous integration service.',
>      keywords: [ 'ci', 'automation', [length]: 2 ],
>      bin: { 'reliable-master': './bin/reliable-master' },
>      repository: 
>       { type: 'git',
>         url: 'git@github.com:reliablejs/reliable-master.git' },
>      dependencies: 
>       { 'ansi-to-html': '~0.3.0',
>         babel: '~5.6.14',
>         badgeboard: '~0.1.1',
>         bcryptjs: '~2.2.1',
>         chalk: '~1.1.1',
>         co: '~4.6',
>         'co-body': '~1.1.0',
>         'co-request': '^0.2.1',
>         commander: '~2.0.0',
>         'detect-port': '~0.1.3',
>         koa: '~0.21.0',
>         'koa-csrf': '2.1.3',
>         'koa-favicon': '~1.2.0',
>         'koa-generic-session': '~1.10.0',
>         'koa-redis': '~1.0.1',
>         'koa-router': '~3.7.0',
>         'koa-static': '~1.4.7',
>         mongoose: '=4.1.5',
>         nodemailer: '~1.4.0',
>         'npm-update': '1.0.2',
>         react: '~0.14.0',
>         'react-dom': '~0.14.2',
>         redis: '~2.3.1',
>         'reliable-events': '^0.1.4',
>         'reliable-github-oauth': '^1.0.0',
>         'reliable-gitlab-oauth': '^1.0.0',
>         'reliable-logger': '^1.0.7',
>         'reliable-mail': '^0.2.0',
>         'reliable-plugin': '^0.1.9',
>         validator: '~3.37.0',
>         xutil: '~1.0.0',
>         yamljs: '~0.2.4',
>         zmq: '2.15.3' },
>      devDependencies: 
>       { 'co-mocha': '~1.1.2',
>         'command-line-test': '^1.0.5',
>         eslint: '~1.1.0',
>         'eslint-plugin-react': '~3.2.3',
>         'git-contributor': '^1.0.8',
>         istanbul: '*',
>         'json-loader': '~0.5.2',
>         'jsx-loader': '~0.13.2',
>         killing: '~1.0.1',
>         mocha: '~2.4.5',
>         'pre-commit': '~1.1.1',
>         'react-d3-components': '~0.6.0',
>         should: '*',
>         webpack: '^1.13.1' },
>      scripts: 
>       { test: 'make test',
>         lint: 'make lint',
>         contributor: 'git-contributor' },
>      'pre-commit': [ 'lint', [length]: 1 ],
>      engines: { node: '>=8.12.0' },
>      homepage: 'https://reliablejs.github.io/reliable',
>      license: 'MIT' } }
> >> index.js:32:10 [worker:4] pid:36 base middlewares attached
> >> index.js:36:12 [worker:4] pid:36 middleware: i18n registed
> >> render.js:15:8 [worker:4] pid:36 render view path: /reliable-master/web/views
> >> index.js:36:12 [worker:4] pid:36 middleware: inject registed
> >> index.js:67:14 [master] pid:19 listening worker id: 5, pid: 42, address: 172.17.0.4:-1
> >> index.js:32:10 [worker:8] pid:60 base middlewares attached
> >> index.js:36:12 [worker:8] pid:60 middleware: i18n registed
> >> render.js:15:8 [worker:8] pid:60 render view path: /reliable-master/web/views
> >> index.js:36:12 [worker:4] pid:36 middleware: favicon registed
> >> index.js:36:12 [worker:8] pid:60 middleware: inject registed
> >> index.js:36:12 [worker:4] pid:36 middleware: powerby registed
> >> index.js:36:12 [worker:8] pid:60 middleware: favicon registed
> >> index.js:36:12 [worker:4] pid:36 middleware: static registed
> >> index.js:36:12 [worker:8] pid:60 middleware: powerby registed
> >> router.js:41:10 [worker:4] pid:36 router set
> >> index.js:36:12 [worker:8] pid:60 middleware: static registed
> >> index.js:67:14 [master] pid:19 listening worker id: 4, pid: 36, address: 172.17.0.4:-1
> >> router.js:41:10 [worker:8] pid:60 router set
> >> index.js:67:14 [master] pid:19 listening worker id: 8, pid: 60, address: 172.17.0.4:-1
> >> Server start at 2018-11-15 17:51:26 | http://172.17.0.4:-1
> >> run.js:89:14 [master] pid:19 slaves all offline with data 
>  { message: 'project' }
> >> manager.js:144:16 [master] pid:19 no slave for monitor #1
> >> run.js:89:14 [master] pid:19 slaves all offline with data 
>  { message: 'project' }
> >> run.js:89:14 [master] pid:19 slaves all offline with data 
>  { message: 'project' }
> >> run.js:89:14 [master] pid:19 slaves all offline with data 
>  { message: 'dispatch' }
> >> manager.js:144:16 [master] pid:19 no slave for monitor #1
> >> run.js:89:14 [master] pid:19 slaves all offline with data 
>  { message: 'project' }
> >> manager.js:144:16 [master] pid:19 no slave for monitor #1
> >> run.js:89:14 [master] pid:19 slaves all offline with data 
>  { message: 'project' }

分析:

1.通过端口查看发现 docker-proxy 使用的协议是 ipv6 协议做的代理映射
2.尝试启用 IPv6 的分组转发,修改 /etc/sysctl.conf: net.ipv6.conf.all.forwarding=1,结果失败~
3.另外尝试服务器禁用 ipv6,重新构建后分配的还是 ipv6,结果失败~
4.最后通过自定义网桥的方式,试着解决该问题,结果还是失败~

写到最后:目前该问题还在探索中,正应了那句话:生命不息,探索不止! 也盼望 macaca 开发团队@xdf等人或踩过类似坑儿的人看到此贴,给予指点帮助,谢谢 !

1. 参考:Port redirecting binding to IPv6 but not IPv4 interfaces

2. 参考:centos7 上如何禁用 ipv6

3. 参考:Docker-网络配置

共收到 2 条回复 时间 点赞
PEPSI 回复

没人回答啊 感觉推广不起来

需要 登录 后方可回复, 如果你还没有账号请点击这里 注册