生产环境Flink的搭建

生产环境用到了Flink,那其实无论腾讯的flink还是aws的flink,都是运行在容器中的

所以我们自建,也用容器,docker-compose.yaml一把梭

 1services:
 2
 3  jobmanager:
 4    image: flink:1.17.1
 5    ports:
 6      - "8081:8081"
 7    command: jobmanager
 8    environment:
 9      - |
10        FLINK_PROPERTIES=
11          jobmanager.memory.process.size: 4gb
12          jobmanager.rpc.address: jobmanager
13
14  taskmanager:
15    image: flink:1.17.1
16    depends_on:
17      - jobmanager
18    command: taskmanager
19    environment:
20      - |
21        FLINK_PROPERTIES=
22        taskmanager.numberOfTaskSlots: 1
23        taskmanager.memory.process.size: 8gb
24        jobmanager.rpc.address: jobmanager
25    deploy:
26          replicas: 1

注意啊,上面节点是16GB内存,所以job分了4G,然后task给了8G,TaskSlot给了1,副本也是1。内存富裕4G。是这么算的

我们再看一个配置,是测试环境的,节点是32G:

 1services:
 2
 3  jobmanager:
 4    image: flink:1.17.1
 5    ports:
 6      - "8081:8081"
 7    command: jobmanager
 8    environment:
 9      - |
10        FLINK_PROPERTIES=
11          jobmanager.memory.process.size: 2gb
12          jobmanager.rpc.address: jobmanager
13
14  taskmanager:
15    image: flink:1.17.1
16    depends_on:
17      - jobmanager
18    command: taskmanager
19    environment:
20      - |
21        FLINK_PROPERTIES=
22        taskmanager.numberOfTaskSlots: 4
23        taskmanager.memory.process.size: 8gb
24        jobmanager.rpc.address: jobmanager
25    deploy:
26          replicas: 1

测试环境的内存就囧多了,jobmanager只给了2G,但是任务多且大,8G和Slot给了4,这个值是不断失败然后修改得到的

我们上生产的时候要根据具体情况来动态调整。


Java进程cpu高排查
comments powered by Disqus