schedulerx 任务分片

       

官网:https://help.aliyun.com/document_detail/148200.html

            

                  

                                

任务分片

        

分片模型:静态分片、动态分片

静态分片:处理固定的分片数,例如分库分表中固定1024张表,需要若干台机器分布式去处理
动态分片:分布式处理未知数据量的数据,例如一张大表在不停变更,需要分布式跑批,暂未开源

           

功能特性

兼容性:兼容elastic-job的静态分片模型
多语言:支持Java、Python、Shell、Go

# 基于Map模型开发,继承了Map模型高可用、流量控制、分片自动失败重试
高可用:某台worker执行过程中发生异常,master worker会把分片failover到其它slave节点执行。
流量控制:控制单机子任务并发度,例如有1000个分片,一共10台机器,
        可以控制最多5个分片并发跑,其它在队列中等待
自动失败重试:某个分片执行子任务失败,可以自动重试

             

                 

                                

使用示例

   

应用1(端口8080)、应用2(端口8081),其余配置及类全部相同

                      

            

application.yml

spring:
  schedulerx2:
    aliyunAccessKey: ***
    aliyunSecretKey: ***
    regionId: public
    endpoint: acm.aliyun.com
    namespaceName: lihu-test
    namespace: ***
    appName: lihu-job
    groupId: lihu-group
    appKey: OOs+XPhRsAa1Ia6+Op+NAw==
    jobs:
      shardingJob:
        jobModel: sharding
        className: com.example.demo.job.CustomShardingJob
        jobParameter: 0=瓜田李下,1=海贼王,2=火影忍者,3=灌篮高手,4=数码宝贝
        cron: "0 */1 * * * ?"

         

DataConfig

@Configuration
public class DataConfig {

    @Bean
    public JobSyncService initJobSyncService(){
        return new JobSyncService();
    }
}

          

CustomSgardingJob

public class CustomShardingJob extends JavaProcessor {

    @Override
    public ProcessResult process(JobContext context) throws Exception {
        System.out.println("分片数:"+context.getShardingNum());
        System.out.println("分片id:"+context.getShardingId());
        System.out.println("分片参数:"+context.getShardingParameter());

        return new ProcessResult(true);
    }
}

        

              

                                

使用测试

   

启动应用1、应用2,控制台查看任务列表

            

            

        

控制台查看任务执行情况

            

              

         

应用1:控制台输出

2022-04-18 14:50:01.053  INFO 1306 --- [er-instance-105] c.a.s.worker.master.MapTaskMaster        : jobInstanceId=716026_998027977, map master config, pageSize:100, queueSize:10000, dispatcherSize:5, workerSize:2
2022-04-18 14:50:01.062  INFO 1306 --- [16026_998027977] c.a.s.worker.master.MapTaskMaster        : socket to 192.168.5.10:50404 is reachable, times=0
2022-04-18 14:50:01.080  INFO 1306 --- [16026_998027977] c.a.s.worker.master.MapTaskMaster        : socket to 192.168.5.10:50384 is reachable, times=0
2022-04-18 14:50:01.080  INFO 1306 --- [026_998027977-0] c.a.s.worker.container.ThreadContainer   : reportTaskStatus instanceId=716026_998027977_0 submitResult=true, processResult=ProcessResult [status=RUNNING, result=null]
2022-04-18 14:50:01.080  INFO 1306 --- [026_998027977-2] c.a.s.worker.container.ThreadContainer   : reportTaskStatus instanceId=716026_998027977_4 submitResult=true, processResult=ProcessResult [status=RUNNING, result=null]
2022-04-18 14:50:01.080  INFO 1306 --- [026_998027977-1] c.a.s.worker.container.ThreadContainer   : reportTaskStatus instanceId=716026_998027977_2 submitResult=true, processResult=ProcessResult [status=RUNNING, result=null]
分片数:5
分片id:4
分片参数:数码宝贝
2022-04-18 14:50:01.096  INFO 1306 --- [026_998027977-2] c.a.s.worker.container.ThreadContainer   : reportTaskStatus instanceId=716026_998027977_4 submitResult=true, processResult=ProcessResult [status=SUCCESS, result=null]
分片数:5
分片id:2
分片参数:火影忍者
分片数:5
分片id:0
分片参数:瓜田李下

        

应用2:控制台输出

2022-04-18 14:49:39.098  INFO 1303 --- [eartbeat-thread] heartbeat                                : heartbeat groupId=lihu-group to akka.tcp://server@112.124.156.1:8080, cost=17ms
2022-04-18 14:49:49.103  INFO 1303 --- [eartbeat-thread] heartbeat                                : heartbeat groupId=lihu-group to akka.tcp://server@112.124.156.1:8080, cost=21ms
2022-04-18 14:49:59.105  INFO 1303 --- [eartbeat-thread] heartbeat                                : heartbeat groupId=lihu-group to akka.tcp://server@112.124.156.1:8080, cost=23ms
2022-04-18 14:50:01.085  INFO 1303 --- [r-container-182] c.a.s.worker.actor.ContainerActor        : jobInstanceId=998027977, batch start containers, size:2
2022-04-18 14:50:01.123  INFO 1303 --- [026_998027977-1] c.a.s.worker.container.ThreadContainer   : reportTaskStatus instanceId=716026_998027977_3 submitResult=true, processResult=ProcessResult [status=RUNNING, result=null]
2022-04-18 14:50:01.123  INFO 1303 --- [026_998027977-0] c.a.s.worker.container.ThreadContainer   : reportTaskStatus instanceId=716026_998027977_1 submitResult=true, processResult=ProcessResult [status=RUNNING, result=null]
分片数:5
分片id:3
分片参数:灌篮高手
分片数:5
分片id:1
分片参数:海贼王