解决flower Substantial drift from.. may mean clocks are out of sync错误
打开flower的监控页面,发现monitor页的succeeded tasks图表始终为空,打印日志发现有下面的警告:
[2016-02-24 12:00:21,799: WARNING/MainProcess] celery@localhost.localdomain ready.
[2016-02-24 12:00:22,297: WARNING/MainProcess] Substantial drift from celery@centos7-181 may mean clocks are out of sync. Current drift is
70 seconds. [orig: 2016-02-24 12:00:22.297798 recv: 2016-02-24 11:59:12.438481]
[2016-02-24 12:00:22,300: WARNING/MainProcess] Substantial drift from celery@centos7-xiaoqiao may mean clocks are out of sync. Current drift is
764 seconds. [orig: 2016-02-24 12:00:22.300171 recv: 2016-02-24 11:47:38.863792]
[2016-02-24 12:00:22,302: WARNING/MainProcess] Substantial drift from celery@centos7-186 may mean clocks are out of sync. Current drift is
65 seconds. [orig: 2016-02-24 12:00:22.302378 recv: 2016-02-24 11:59:17.157844]
[2016-02-24 12:00:22,303: WARNING/MainProcess] Substantial drift from celery@centos7-182 may mean clocks are out of sync. Current drift is
70 seconds. [orig: 2016-02-24 12:00:22.303616 recv: 2016-02-24 11:59:12.633749]
[2016-02-24 12:00:22,306: WARNING/MainProcess] Substantial drift from celery@centos7-189 may mean clocks are out of sync. Current drift is
64 seconds. [orig: 2016-02-24 12:00:22.306499 recv: 2016-02-24 11:59:18.100351]
[2016-02-24 12:00:22,307: WARNING/MainProcess] Substantial drift from celery@centos7-188 may mean clocks are out of sync. Current drift is
65 seconds. [orig: 2016-02-24 12:00:22.307066 recv: 2016-02-24 11:59:17.197479]
[2016-02-24 12:00:22,310: WARNING/MainProcess] Substantial drift from celery@centos7-184 may mean clocks are out of sync. Current drift is
64 seconds. [orig: 2016-02-24 12:00:22.309941 recv: 2016-02-24 11:59:18.490792]
[2016-02-24 12:00:22,524: WARNING/MainProcess] Substantial drift from celery@centos7-183 may mean clocks are out of sync. Current drift is
64 seconds. [orig: 2016-02-24 12:00:22.524618 recv: 2016-02-24 11:59:18.005931]
[2016-02-24 12:00:22,525: WARNING/MainProcess] Substantial drift from celery@centos7-187 may mean clocks are out of sync. Current drift is
70 seconds. [orig: 2016-02-24 12:00:22.525161 recv: 2016-02-24 11:59:12.294425]
[2016-02-24 12:00:22,525: WARNING/MainProcess] Substantial drift from celery@centos7-185 may mean clocks are out of sync. Current drift is
65 seconds. [orig: 2016-02-24 12:00:22.525489 recv: 2016-02-24 11:59:17.309409]
字面看是由于时间不同步造成的,可是在settings里已经设置了USE_TZ = True
以及相应的时区选项,经过一番费力的寻找发现造成这个问题的原因居然是每台机器的时间不同!至于为什么机器设置为同一时区却有时间差就不探究了,解决起来就是把各个机器的时间同步就可以了:
yum install ntpdate -y
ntpdate 0.asia.pool.ntp.org
网上很多Ntp服务器都失效了,可以使用下面的:
0.asia.pool.ntp.org
1.asia.pool.ntp.org
2.asia.pool.ntp.org
3.asia.pool.ntp.org
另外最好设置个计划任务定时去同步。