生产环境用到了Flink,那其实无论腾讯的flink还是aws的flink,都是运行在容器中的

所以我们自建,也用容器,docker-compose.yaml一把梭

services:

  jobmanager:
    image: flink:1.17.1
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      - |
        FLINK_PROPERTIES=
          jobmanager.memory.process.size: 4gb
          jobmanager.rpc.address: jobmanager

  taskmanager:
    image: flink:1.17.1
    depends_on:
      - jobmanager
    command: taskmanager
    environment:
      - |
        FLINK_PROPERTIES=
        taskmanager.numberOfTaskSlots: 1
        taskmanager.memory.process.size: 8gb
        jobmanager.rpc.address: jobmanager
    deploy:
          replicas: 1

注意啊,上面节点是16GB内存,所以job分了4G,然后task给了8G,TaskSlot给了1,副本也是1。内存富裕4G。是这么算的

我们再看一个配置,是测试环境的,节点是32G:

services:

  jobmanager:
    image: flink:1.17.1
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      - |
        FLINK_PROPERTIES=
          jobmanager.memory.process.size: 2gb
          jobmanager.rpc.address: jobmanager

  taskmanager:
    image: flink:1.17.1
    depends_on:
      - jobmanager
    command: taskmanager
    environment:
      - |
        FLINK_PROPERTIES=
        taskmanager.numberOfTaskSlots: 4
        taskmanager.memory.process.size: 8gb
        jobmanager.rpc.address: jobmanager
    deploy:
          replicas: 1

测试环境的内存就囧多了,jobmanager只给了2G,但是任务多且大,8G和Slot给了4,这个值是不断失败然后修改得到的

我们上生产的时候要根据具体情况来动态调整。