On Astro Hybrid clusters, worker type is defined as a node instance type that is supported by the cloud provider of your cluster. Worker TypeĪll worker types additionally have 10 GiB of ephemeral storage that your tasks can use when storing small amounts of data within the worker. The following table lists all available worker types on Astro Deployments. Celery workers can run multiple tasks at once, while Kubernetes workers only scale up and down to run a single task at a time. The underlying node instance type running your worker can vary based on how Astro optimizes resource usage on your cluster.Įach virtualized instance of your worker type is a worker. A worker type is a virtualized instance of CPU and memory on your cluster that is specific to the Astro platform. For more information, see the following section on hosted worker types.Įach Deployment worker queue has a worker type that determines how many resources are available to your Airflow workers for running tasks. A worker’s total available CPU, memory, and storage is defined by its worker size.
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |