生产环境用到了Flink,那其实无论腾讯的flink还是aws的flink,都是运行在容器中的
所以我们自建,也用容器,docker-compose.yaml一把梭
1services:
2
3 jobmanager:
4 image: flink:1.17.1
5 ports:
6 - "8081:8081"
7 command: jobmanager
8 environment:
9 - |
10 FLINK_PROPERTIES=
11 jobmanager.memory.process.size: 4gb
12 jobmanager.rpc.address: jobmanager
13
14 taskmanager:
15 image: flink:1.17.1
16 depends_on:
17 - jobmanager
18 command: taskmanager
19 environment:
20 - |
21 FLINK_PROPERTIES=
22 taskmanager.numberOfTaskSlots: 1
23 taskmanager.memory.process.size: 8gb
24 jobmanager.rpc.address: jobmanager
25 deploy:
26 replicas: 1
注意啊,上面节点是16GB内存,所以job分了4G,然后task给了8G,TaskSlot给了1,副本也是1。内存富裕4G。是这么算的
我们再看一个配置,是测试环境的,节点是32G:
1services:
2
3 jobmanager:
4 image: flink:1.17.1
5 ports:
6 - "8081:8081"
7 command: jobmanager
8 environment:
9 - |
10 FLINK_PROPERTIES=
11 jobmanager.memory.process.size: 2gb
12 jobmanager.rpc.address: jobmanager
13
14 taskmanager:
15 image: flink:1.17.1
16 depends_on:
17 - jobmanager
18 command: taskmanager
19 environment:
20 - |
21 FLINK_PROPERTIES=
22 taskmanager.numberOfTaskSlots: 4
23 taskmanager.memory.process.size: 8gb
24 jobmanager.rpc.address: jobmanager
25 deploy:
26 replicas: 1
测试环境的内存就囧多了,jobmanager只给了2G,但是任务多且大,8G和Slot给了4,这个值是不断失败然后修改得到的
我们上生产的时候要根据具体情况来动态调整。