dask-root-397889d9-64d5zf dask-worker [2020-05-08 09:05:38] DEBUG - prefect.CloudTaskRunner | Task 'SaveResponse[363]': Handling state change from Running to Success dask-root-397889d9-64d5zf dask-worker [2020-05-08 09:05:38] DEBUG - iguazu.core.handlers | Managing garbage collection for SaveResponse on change from Running to Success dask-root-397889d9-64d5zf dask-worker [2020-05-08 09:05:38] DEBUG - iguazu.core.handlers | Memory usage is 0.15 Gb (1.03 %) dask-root-397889d9-64d5zf dask-worker [2020-05-08 09:05:38] DEBUG - iguazu.core.handlers | Iguazu logging handler is maybe deprecated now? dask-root-397889d9-6h9bvs dask-worker [2020-05-08 09:05:38] DEBUG - iguazu.core.tasks | Managing auto_clean for task dask-root-397889d9-6h9bvs dask-worker [2020-05-08 09:05:38] DEBUG - iguazu.core.tasks | Task does not have auto_clean_files enabled dask-root-397889d9-6h9bvs dask-worker [2020-05-08 09:05:38] DEBUG - prefect.CloudTaskRunner | Task 'SaveResponse[381]': Handling state change from Running to Success dask-root-397889d9-6h9bvs dask-worker [2020-05-08 09:05:38] DEBUG - iguazu.core.handlers | Managing garbage collection for SaveResponse on change from Running to Success dask-root-397889d9-6h9bvs dask-worker [2020-05-08 09:05:38] DEBUG - iguazu.core.handlers | Memory usage is 0.15 Gb (1.05 %) dask-root-397889d9-6h9bvs dask-worker [2020-05-08 09:05:38] DEBUG - iguazu.core.handlers | Iguazu logging handler is maybe deprecated now? dask-root-397889d9-6z9s8c dask-worker [2020-05-08 09:05:38] INFO - prefect.CloudTaskRunner | Task 'SaveResponse[386]': finished task run for task with final state: 'Success' dask-root-397889d9-6vx5x8 dask-worker [2020-05-08 09:05:38] DEBUG - iguazu.core.tasks | Managing auto_clean for task dask-root-397889d9-6vx5x8 dask-worker [2020-05-08 09:05:38] DEBUG - iguazu.core.tasks | Task does not have auto_clean_files enabled dask-root-397889d9-6vx5x8 dask-worker [2020-05-08 09:05:38] DEBUG - prefect.CloudTaskRunner | Task 'SaveResponse[357]': Handling state change from Running to Success dask-root-397889d9-6vx5x8 dask-worker [2020-05-08 09:05:38] DEBUG - iguazu.core.handlers | Managing garbage collection for SaveResponse on change from Running to Success dask-root-397889d9-6vx5x8 dask-worker [2020-05-08 09:05:38] DEBUG - iguazu.core.handlers | Memory usage is 0.16 Gb (1.07 %) dask-root-397889d9-6vx5x8 dask-worker [2020-05-08 09:05:38] DEBUG - iguazu.core.handlers | Iguazu logging handler is maybe deprecated now? dask-root-397889d9-64d5zf dask-worker [2020-05-08 09:05:38] INFO - prefect.CloudTaskRunner | Task 'SaveResponse[363]': finished task run for task with final state: 'Success' dask-root-397889d9-6h9bvs dask-worker [2020-05-08 09:05:38] INFO - prefect.CloudTaskRunner | Task 'SaveResponse[381]': finished task run for task with final state: 'Success' prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Closing worker tcp://10.16.1.16:37649 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Remove worker prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.core - INFO - Removing comms to tcp://10.16.1.16:37649 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v dask-root-397889d9-6prjns flow distributed.scheduler - INFO - Closing worker tcp://10.16.1.11:40537 dask-worker distributed.worker - INFO - Stopping worker at tcp://10.16.1.11:40537 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Remove worker prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.core - INFO - Removing comms to tcp://10.16.1.11:40537 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Closing worker tcp://10.16.1.18:45395 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Remove worker prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.core - INFO - Removing comms to tcp://10.16.1.18:45395 dask-root-397889d9-6x4xcm dask-worker distributed.worker - INFO - Stopping worker at tcp://10.16.1.16:37649 dask-root-397889d9-6f76d6 dask-worker distributed.worker - INFO - Stopping worker at tcp://10.16.1.18:45395 dask-root-397889d9-6vx5x8 dask-worker [2020-05-08 09:05:38] INFO - prefect.CloudTaskRunner | Task 'SaveResponse[357]': finished task run for task with final state: 'Success' prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.deploy.adaptive - INFO - Retiring workers [34, 27, 38] prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow [2020-05-08 09:05:38] INFO - distributed.deploy.adaptive | Retiring workers [34, 27, 38] dask-root-397889d9-6f76d6 dask-worker distributed.dask_worker - INFO - Exiting on signal 15 dask-root-397889d9-6f76d6 dask-worker distributed.nanny - INFO - Closing Nanny at 'tcp://10.16.1.18:42881' dask-root-397889d9-6x4xcm dask-worker distributed.dask_worker - INFO - Exiting on signal 15 dask-root-397889d9-6prjns dask-worker distributed.dask_worker - INFO - Exiting on signal 15 dask-root-397889d9-6x4xcm dask-worker distributed.nanny - INFO - Closing Nanny at 'tcp://10.16.1.16:34783' dask-root-397889d9-6prjns dask-worker distributed.nanny - INFO - Closing Nanny at 'tcp://10.16.1.11:45367' prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Retire worker names (32, 30, 39) prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Retire workers {, , } prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Moving 91 keys to other workers dask-root-397889d9-6f76d6 dask-worker distributed.nanny - INFO - Worker closed dask-root-397889d9-6x4xcm dask-worker distributed.nanny - INFO - Worker closed dask-root-397889d9-6prjns dask-worker distributed.nanny - INFO - Worker closed prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.utils - ERROR - prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow Traceback (most recent call last): prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow File "/opt/pysetup/.venv/lib/python3.8/site-packages/distributed/utils.py", line 664, in log_errors prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow yield prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow File "/opt/pysetup/.venv/lib/python3.8/site-packages/distributed/scheduler.py", line 3455, in retire_workers prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow await self.replicate( prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow File "/opt/pysetup/.venv/lib/python3.8/site-packages/distributed/scheduler.py", line 3214, in replicate prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow assert count > 0 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow AssertionError prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.core - ERROR - prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow Traceback (most recent call last): prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow File "/opt/pysetup/.venv/lib/python3.8/site-packages/distributed/core.py", line 403, in handle_comm prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow result = await result prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow File "/opt/pysetup/.venv/lib/python3.8/site-packages/distributed/scheduler.py", line 3455, in retire_workers prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow await self.replicate( prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow File "/opt/pysetup/.venv/lib/python3.8/site-packages/distributed/scheduler.py", line 3214, in replicate prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow assert count > 0 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow AssertionError dask-root-397889d9-62vzh7 dask-worker [2020-05-08 09:05:40] DEBUG - prefect.CloudTaskRunner | 399 mapped tasks submitted for execution. dask-root-397889d9-62vzh7 dask-worker [2020-05-08 09:05:40] DEBUG - prefect.CloudTaskRunner | Task 'SaveResponse': Handling state change from Mapped to Mapped dask-root-397889d9-62vzh7 dask-worker [2020-05-08 09:05:40] DEBUG - iguazu.core.handlers | Iguazu logging handler is maybe deprecated now? prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.utils - ERROR - prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow Traceback (most recent call last): prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow File "/opt/pysetup/.venv/lib/python3.8/site-packages/distributed/utils.py", line 664, in log_errors prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow yield prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow File "/opt/pysetup/.venv/lib/python3.8/site-packages/distributed/deploy/adaptive.py", line 185, in scale_down prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow await self.scheduler.retire_workers( prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow File "/opt/pysetup/.venv/lib/python3.8/site-packages/distributed/core.py", line 679, in send_recv_from_rpc prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow result = await send_recv(comm=comm, op=key, **kwargs) prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow File "/opt/pysetup/.venv/lib/python3.8/site-packages/distributed/core.py", line 549, in send_recv prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow raise exc.with_traceback(tb) prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow File "/opt/pysetup/.venv/lib/python3.8/site-packages/distributed/core.py", line 403, in handle_comm prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow result = await result prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow File "/opt/pysetup/.venv/lib/python3.8/site-packages/distributed/scheduler.py", line 3455, in retire_workers prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow await self.replicate( prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow File "/opt/pysetup/.venv/lib/python3.8/site-packages/distributed/scheduler.py", line 3214, in replicate prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow assert count > 0 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow AssertionError prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow tornado.application - ERROR - Exception in callback functools.partial(>, exception=AssertionError()>) prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow Traceback (most recent call last): prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow File "/opt/pysetup/.venv/lib/python3.8/site-packages/tornado/ioloop.py", line 743, in _run_callback prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow ret = callback() prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow File "/opt/pysetup/.venv/lib/python3.8/site-packages/tornado/ioloop.py", line 767, in _discard_future_result prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow future.result() prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow File "/opt/pysetup/.venv/lib/python3.8/site-packages/distributed/deploy/adaptive_core.py", line 194, in adapt prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow await self.scale_down(**recommendations) prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow File "/opt/pysetup/.venv/lib/python3.8/site-packages/distributed/deploy/adaptive.py", line 185, in scale_down prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow await self.scheduler.retire_workers( prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow File "/opt/pysetup/.venv/lib/python3.8/site-packages/distributed/core.py", line 679, in send_recv_from_rpc prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow result = await send_recv(comm=comm, op=key, **kwargs) prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow File "/opt/pysetup/.venv/lib/python3.8/site-packages/distributed/core.py", line 549, in send_recv prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow raise exc.with_traceback(tb) prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow File "/opt/pysetup/.venv/lib/python3.8/site-packages/distributed/core.py", line 403, in handle_comm prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow result = await result prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow File "/opt/pysetup/.venv/lib/python3.8/site-packages/distributed/scheduler.py", line 3455, in retire_workers prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow await self.replicate( prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow File "/opt/pysetup/.venv/lib/python3.8/site-packages/distributed/scheduler.py", line 3214, in replicate prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow assert count > 0 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow AssertionError dask-root-397889d9-62vzh7 dask-worker [2020-05-08 09:05:40] DEBUG - prefect.CloudTaskRunner | Task 'SaveResponse': task has been mapped; ending run. dask-root-397889d9-62vzh7 dask-worker [2020-05-08 09:05:40] INFO - prefect.CloudTaskRunner | Task 'SaveResponse': finished task run for task with final state: 'Mapped' dask-root-397889d9-6f76d6 dask-worker distributed.nanny - WARNING - Worker process still alive after 1 seconds, killing dask-root-397889d9-6f76d6 dask-root-397889d9-6x4xcm dask-worker distributed.nanny - WARNING - Worker process still alive after 1 seconds, killing dask-worker distributed.dask_worker - INFO - End worker dask-root-397889d9-6f76d6 dask-worker distributed.process - INFO - reaping stray process dask-root-397889d9-6x4xcm dask-worker distributed.dask_worker - INFO - End worker dask-root-397889d9-6x4xcm dask-worker distributed.process - INFO - reaping stray process dask-root-397889d9-6prjns dask-worker distributed.nanny - WARNING - Worker process still alive after 1 seconds, killing dask-root-397889d9-6prjns dask-worker distributed.dask_worker - INFO - End worker dask-root-397889d9-6prjns dask-worker distributed.process - INFO - reaping stray process prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Retire worker names (33, 35, 36, 31) dask-root-397889d9-62vzh7 dask-worker [2020-05-08 09:05:40] INFO - prefect.CloudTaskRunner | Task 'Report': Starting task run... dask-root-397889d9-62vzh7 dask-worker [2020-05-08 09:05:40] DEBUG - prefect.CloudTaskRunner | Task 'Report': not all upstream states are finished; ending run. dask-root-397889d9-62vzh7 dask-worker [2020-05-08 09:05:40] INFO - prefect.CloudTaskRunner | Task 'Report': finished task run for task with final state: 'Pending' prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Retire workers {, , , } prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v dask-root-397889d9-69s2kw dask-worker distributed.worker - INFO - Stopping worker at tcp://10.16.2.13:34477 dask-root-397889d9-6hhznr dask-worker distributed.worker - INFO - Stopping worker at tcp://10.16.2.12:35361 flow distributed.scheduler - INFO - Closing worker tcp://10.16.2.12:35361 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Remove worker prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.core - INFO - Removing comms to tcp://10.16.2.12:35361 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Closing worker tcp://10.16.2.15:41209 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Remove worker prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.core - INFO - Removing comms to tcp://10.16.2.15:41209 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Closing worker tcp://10.16.2.13:34477 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Remove worker prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.core - INFO - Removing comms to tcp://10.16.2.13:34477 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Closing worker tcp://10.16.2.10:35507 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Remove worker prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.core - INFO - Removing comms to tcp://10.16.2.10:35507 dask-root-397889d9-64w7n7 dask-worker distributed.worker - INFO - Stopping worker at tcp://10.16.2.15:41209 dask-root-397889d9-65qz68 dask-worker distributed.worker - INFO - Stopping worker at tcp://10.16.2.10:35507 dask-root-397889d9-62vzh7 dask-worker [2020-05-08 09:05:41] INFO - prefect.CloudTaskRunner | Task 'SlackTask': Starting task run... dask-root-397889d9-62vzh7 dask-worker [2020-05-08 09:05:41] DEBUG - prefect.CloudTaskRunner | Task 'SlackTask': not all upstream states are finished; ending run. dask-root-397889d9-62vzh7 dask-worker [2020-05-08 09:05:41] INFO - prefect.CloudTaskRunner | Task 'SlackTask': finished task run for task with final state: 'Pending' prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow [2020-05-08 09:05:40] INFO - distributed.deploy.adaptive | Retiring workers [33, 35, 36, 31] prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.deploy.adaptive - INFO - Retiring workers [33, 35, 36, 31] dask-root-397889d9-6hhznr dask-worker distributed.dask_worker - INFO - Exiting on signal 15 dask-root-397889d9-6hhznr dask-worker distributed.nanny - INFO - Closing Nanny at 'tcp://10.16.2.12:34309' dask-root-397889d9-69s2kw dask-root-397889d9-65qz68 dask-worker distributed.dask_worker - INFO - Exiting on signal 15 dask-worker distributed.dask_worker - INFO - Exiting on signal 15 dask-root-397889d9-69s2kw dask-worker distributed.nanny - INFO - Closing Nanny at 'tcp://10.16.2.13:33697' dask-root-397889d9-65qz68 dask-worker distributed.nanny - INFO - Closing Nanny at 'tcp://10.16.2.10:41645' prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Remove client Client-65ae0be6-910a-11ea-8006-32a282f501e9 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Remove client Client-65ae0be6-910a-11ea-8006-32a282f501e9 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow [2020-05-08 09:05:41] INFO - prefect.CloudFlowRunner | Flow run RUNNING: terminal tasks are incomplete. dask-root-397889d9-64w7n7 dask-worker distributed.dask_worker - INFO - Exiting on signal 15 dask-root-397889d9-64w7n7 dask-worker distributed.nanny - INFO - Closing Nanny at 'tcp://10.16.2.15:34701' prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Close client connection: Client-65ae0be6-910a-11ea-8006-32a282f501e9 dask-root-397889d9-69s2kw dask-worker distributed.nanny - INFO - Worker closed dask-root-397889d9-65qz68 dask-worker distributed.nanny - INFO - Worker closed dask-root-397889d9-64w7n7dask-root-397889d9-6hhznr dask-worker distributed.nanny - INFO - Worker closed dask-worker distributed.nanny - INFO - Worker closed prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Retire worker names (40, 28, 29) dask-root-397889d9-6vx5x8prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v dask-root-397889d9-64d5zf flow distributed.scheduler - INFO - Retire workers {, , } dask-workerprefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v distributed.worker - INFO - Stopping worker at tcp://10.16.1.15:41397 dask-worker distributed.worker - INFO - Stopping worker at tcp://10.16.1.12:45731 flow distributed.scheduler - INFO - Closing worker tcp://10.16.1.12:45731 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Remove worker dask-root-397889d9-6h9bvs dask-worker distributed.worker - INFO - Stopping worker at tcp://10.16.1.10:34847 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.core - INFO - Removing comms to tcp://10.16.1.12:45731 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Closing worker tcp://10.16.1.10:34847 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Remove worker prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.core - INFO - Removing comms to tcp://10.16.1.10:34847 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Closing worker tcp://10.16.1.15:41397 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Remove worker prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.core - INFO - Removing comms to tcp://10.16.1.15:41397 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow [2020-05-08 09:05:42] INFO - distributed.deploy.adaptive | Retiring workers [40, 28, 29] prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.deploy.adaptive - INFO - Retiring workers [40, 28, 29] dask-root-397889d9-64d5zf dask-worker distributed.dask_worker - INFO - Exiting on signal 15 dask-root-397889d9-64d5zf dask-worker distributed.nanny - INFO - Closing Nanny at 'tcp://10.16.1.15:36951' dask-root-397889d9-6h9bvs dask-worker distributed.dask_worker - INFO - Exiting on signal 15 dask-root-397889d9-6h9bvs dask-worker distributed.nanny - INFO - Closing Nanny at 'tcp://10.16.1.10:37161' dask-root-397889d9-6vx5x8 dask-worker distributed.dask_worker - INFO - Exiting on signal 15 dask-root-397889d9-6vx5x8 dask-worker distributed.nanny - INFO - Closing Nanny at 'tcp://10.16.1.12:38971' dask-root-397889d9-6hhznr dask-worker distributed.nanny - WARNING - Worker process still alive after 1 seconds, killing dask-root-397889d9-6hhznr dask-worker distributed.dask_worker - INFO - End worker dask-root-397889d9-6hhznr dask-worker distributed.process - INFO - reaping stray process dask-root-397889d9-69s2kw dask-worker distributed.nanny - WARNING - Worker process still alive after 1 seconds, killing dask-root-397889d9-69s2kw dask-worker distributed.dask_worker - INFO - End worker dask-root-397889d9-69s2kw dask-worker distributed.process - INFO - reaping stray process dask-root-397889d9-65qz68 dask-worker distributed.nanny - WARNING - Worker process still alive after 1 seconds, killing dask-root-397889d9-65qz68 dask-worker distributed.dask_worker - INFO - End worker dask-root-397889d9-65qz68 dask-worker distributed.process - INFO - reaping stray process dask-root-397889d9-64w7n7 dask-worker distributed.nanny - WARNING - Worker process still alive after 1 seconds, killing dask-root-397889d9-64w7n7 dask-worker distributed.dask_worker - INFO - End worker dask-root-397889d9-64w7n7 dask-worker distributed.process - INFO - reaping stray process dask-root-397889d9-64w7n7 dask-worker Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join: dask-root-397889d9-64w7n7 dask-worker Traceback (most recent call last): dask-root-397889d9-64w7n7 dask-worker File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner dask-root-397889d9-64w7n7 dask-worker self.run() dask-root-397889d9-64w7n7 dask-worker File "/usr/local/lib/python3.8/threading.py", line 870, in run dask-root-397889d9-64w7n7 dask-worker self._target(*self._args, **self._kwargs) dask-root-397889d9-64w7n7 dask-worker File "/opt/pysetup/.venv/lib/python3.8/site-packages/distributed/process.py", line 234, in _watch_process prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Retire worker names (32, 37, 30, 39) prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Retire workers {, , , } prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Closing worker tcp://10.16.2.14:36189 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Remove worker prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.core - INFO - Removing comms to tcp://10.16.2.14:36189 dask-root-397889d9-6z9s8c dask-worker distributed.worker - INFO - Stopping worker at tcp://10.16.2.14:36189 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Closing worker tcp://10.16.2.11:41147 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Remove worker prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.core - INFO - Removing comms to tcp://10.16.2.11:41147 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Closing worker tcp://10.16.1.17:40839 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Remove worker prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.core - INFO - Removing comms to tcp://10.16.1.17:40839 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Closing worker tcp://10.16.2.9:36489 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Remove worker prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.core - INFO - Removing comms to tcp://10.16.2.9:36489 dask-root-397889d9-6vx5x8 dask-worker distributed.nanny - INFO - Worker closed dask-root-397889d9-6h9bvs dask-worker distributed.nanny - INFO - Worker closed dask-root-397889d9-64d5zf dask-worker distributed.nanny - INFO - Worker closed dask-root-397889d9-6mj6rm dask-worker distributed.worker - INFO - Stopping worker at tcp://10.16.1.17:40839 dask-root-397889d9-6ght7d dask-worker distributed.worker - INFO - Stopping worker at tcp://10.16.2.11:41147 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow [2020-05-08 09:05:43] INFO - distributed.deploy.adaptive | Retiring workers [32, 37, 30, 39] prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow dask-root-397889d9-6pgjj4 distributed.deploy.adaptive - INFO - Retiring workers [32, 37, 30, 39] dask-worker distributed.worker - INFO - Stopping worker at tcp://10.16.2.9:36489 dask-root-397889d9-6mj6rm dask-worker distributed.dask_worker - INFO - Exiting on signal 15 dask-root-397889d9-6ght7d dask-root-397889d9-6mj6rm dask-worker distributed.nanny - INFO - Closing Nanny at 'tcp://10.16.1.17:42935' dask-worker distributed.dask_worker - INFO - Exiting on signal 15 dask-root-397889d9-6ght7d dask-worker distributed.nanny - INFO - Closing Nanny at 'tcp://10.16.2.11:37267' dask-root-397889d9-6z9s8c dask-worker distributed.dask_worker - INFO - Exiting on signal 15 dask-root-397889d9-6z9s8c dask-worker dask-root-397889d9-6pgjj4 dask-worker distributed.dask_worker - INFO - Exiting on signal 15 distributed.nanny - INFO - Closing Nanny at 'tcp://10.16.2.14:42075' dask-root-397889d9-6pgjj4 dask-worker distributed.nanny - INFO - Closing Nanny at 'tcp://10.16.2.9:45803' dask-root-397889d9-64d5zf dask-worker distributed.nanny - WARNING - Worker process still alive after 1 seconds, killing dask-root-397889d9-64d5zf dask-worker distributed.dask_worker - INFO - End worker dask-root-397889d9-64d5zf dask-worker distributed.process - INFO - reaping stray process dask-root-397889d9-6vx5x8 dask-worker distributed.nanny - WARNING - Worker process still alive after 1 seconds, killing dask-root-397889d9-6h9bvs dask-worker distributed.nanny - WARNING - Worker process still alive after 1 seconds, killing dask-root-397889d9-6h9bvs dask-worker distributed.dask_worker - INFO - End worker dask-root-397889d9-6h9bvs dask-worker distributed.process - INFO - reaping stray process dask-root-397889d9-6vx5x8 dask-worker distributed.dask_worker - INFO - End worker dask-root-397889d9-6vx5x8 dask-worker distributed.process - INFO - reaping stray process dask-root-397889d9-6z9s8c dask-worker distributed.nanny - INFO - Worker closed dask-root-397889d9-6mj6rm dask-worker distributed.nanny - INFO - Worker closed dask-root-397889d9-6ght7d dask-worker distributed.nanny - INFO - Worker closed dask-root-397889d9-6pgjj4 dask-worker distributed.nanny - INFO - Worker closed dask-root-397889d9-6mj6rm dask-worker distributed.nanny - WARNING - Worker process still alive after 1 seconds, killing dask-root-397889d9-6z9s8c dask-workerdask-root-397889d9-6mj6rm distributed.nanny - WARNING - Worker process still alive after 1 seconds, killing dask-root-397889d9-6ght7d dask-root-397889d9-6z9s8c dask-worker distributed.dask_worker - INFO - End worker dask-root-397889d9-6z9s8c dask-worker dask-worker distributed.dask_worker - INFO - End worker dask-worker dask-root-397889d9-6mj6rm distributed.nanny - WARNING - Worker process still alive after 1 seconds, killing dask-root-397889d9-6ght7d dask-worker distributed.dask_worker - INFO - End worker dask-root-397889d9-6ght7d dask-worker distributed.process - INFO - reaping stray process dask-worker distributed.process - INFO - reaping stray process distributed.process - INFO - reaping stray process dask-root-397889d9-6pgjj4 dask-worker distributed.nanny - WARNING - Worker process still alive after 1 seconds, killing dask-root-397889d9-6pgjj4 dask-worker distributed.dask_worker - INFO - End worker dask-root-397889d9-6pgjj4 dask-worker distributed.process - INFO - reaping stray process dask-root-397889d9-6h5lg5 dask-worker distributed.dask_worker - INFO - Exiting on signal 15 dask-root-397889d9-6h5lg5 dask-worker distributed.nanny - INFO - Closing Nanny at 'tcp://10.16.0.40:46871' dask-root-397889d9-62vzh7 dask-worker distributed.dask_worker - INFO - Exiting on signal 15 dask-root-397889d9-62vzh7 dask-worker distributed.nanny - INFO - Closing Nanny at 'tcp://10.16.0.38:37099' dask-root-397889d9-6h5lg5 dask-worker dask-root-397889d9-62vzh7 distributed.worker - INFO - Stopping worker at tcp://10.16.0.40:38917 dask-root-397889d9-6h5lg5 dask-worker distributed.worker - WARNING - Heartbeat to scheduler failed dask-worker distributed.worker - INFO - Stopping worker at tcp://10.16.0.38:35899 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Remove worker prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.core - INFO - Removing comms to tcp://10.16.0.40:38917 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Remove worker prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.core - INFO - Removing comms to tcp://10.16.0.38:35899 prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Lost all workers prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Scheduler closing... prefect-dask-job-4d594928-3def-45e5-bf6a-4237dfa9e824-w4x7v flow distributed.scheduler - INFO - Scheduler closing all comms dask-root-397889d9-62vzh7 dask-worker distributed.nanny - INFO - Worker closed dask-root-397889d9-6h5lg5 dask-worker distributed.nanny - INFO - Worker closed