生产环境用到了Flink,那其实无论腾讯的flink还是aws的flink,都是运行在容器中的
所以我们自建,也用容器,docker-compose.yaml一把梭
services:
jobmanager:
image: flink:1.17.1
ports:
- "8081:8081"
command: jobmanager
environment:
- |
FLINK_PROPERTIES=
jobmanager.memory.process.size: 4gb
jobmanager.rpc.address: jobmanager
taskmanager:
image: flink:1.17.1
depends_on:
- jobmanager
command: taskmanager
environment:
- |
FLINK_PROPERTIES=
taskmanager.numberOfTaskSlots: 1
taskmanager.memory.process.size: 8gb
jobmanager.rpc.address: jobmanager
deploy:
replicas: 1
注意啊,上面节点是16GB内存,所以job分了4G,然后task给了8G,TaskSlot给了1,副本也是1。内存富裕4G。是这么算的
我们再看一个配置,是测试环境的,节点是32G:
services:
jobmanager:
image: flink:1.17.1
ports:
- "8081:8081"
command: jobmanager
environment:
- |
FLINK_PROPERTIES=
jobmanager.memory.process.size: 2gb
jobmanager.rpc.address: jobmanager
taskmanager:
image: flink:1.17.1
depends_on:
- jobmanager
command: taskmanager
environment:
- |
FLINK_PROPERTIES=
taskmanager.numberOfTaskSlots: 4
taskmanager.memory.process.size: 8gb
jobmanager.rpc.address: jobmanager
deploy:
replicas: 1
测试环境的内存就囧多了,jobmanager只给了2G,但是任务多且大,8G和Slot给了4,这个值是不断失败然后修改得到的
我们上生产的时候要根据具体情况来动态调整。