-
Notifications
You must be signed in to change notification settings - Fork 64
leasing pull tasks
Pull Queues is not supported for this runtime.
Once tasks are in a pull queue, a worker can lease them. After the tasks are processed the worker must delete them.
- Create a pull queue.
- Create tasks and add them to the pull queue.
- This method is only applicable to workers that are running within a service in the standard environment.
- When you use pull queues, you are responsible for scaling your workers based on your processing volume.
After the tasks are in the queue, a worker can lease one or more of them using the
lease_tasks()
method. There may be a short delay before tasks recently added using
add()
add()
become available via
lease_tasks()
When you request a lease, you specify the number of tasks to lease (up to a
maximum of 1,000 tasks) and the duration of the lease in seconds (up to a
maximum of one week). The lease duration needs to be long enough to ensure that
the slowest task will have time to finish before the lease period expires. You
can modify a task lease using
modifyTaskLease()
Leasing a task makes it unavailable for processing by another worker, and it remains unavailable until the lease expires.
The
lease_tasks()
lease_tasks()
method returns a
Task
object containing a list of tasks leased from the queue.
Note: taskqueue.Lease
lease_tasks() operates only on pull queues. If you attempt to lease tasks
added in a push queue, App Engine returns an errorthrows an exception.
The following code sample leases tasks from the queue pull-queue for one hour:
View TaskqueueServlet.java on GitHub (region:
lease_tasks)
Not all tasks are alike; your code can "tag" tasks and then choose tasks to lease by tag. The tag acts as a filter.
View TaskqueueServlet.java on GitHub (region:
add_task_w_tag)
Then lease the filtered tasks:
View TaskqueueServlet.java on GitHub (region:
lease_tasks_by_tag)
The following code sample leases 100 tasks from the queue pull-queue for one
hour:
Not all tasks are alike; your code can "tag" tasks and then choose tasks to lease by tag. The tag acts as a filter. The following code sample demonstrates how to tag tasks and then lease by tags:
The following code sample leases 100 tasks from the queue pull-queue for one
hour:
Not all tasks are alike; your code can "tag" tasks and then choose tasks to lease by tag. The tag acts as a filter.
Workers that poll the queue for tasks to lease should detect whether they are
attempting to lease tasks faster than the queue can supply them. If this failure
occurs, a back-off error will be returned from taskqueue.Lease. Your code must
handle these errors, back off from calling taskqueue.Lease, and then try again
later. To avoid this problem, consider setting a higher RPC deadline when
calling taskqueue.Lease. You should also back off when a lease request returns
an empty list of tasks. If you generate more than 10 LeaseTasks requests per
queue per second, only the first 10 requests will return results. If requests
exceed this limit, OK is returned with zero results.
Workers that poll the queue for tasks to lease should detect whether they are
attempting to lease tasks faster than the queue can supply them. If this failure
occurs, the following exceptions from
lease_tasks()
lease_tasks()
can be generated:
- `google.appengine.api.taskqueue.TransientError`
- `google.appengine.runtime.apiproxy_errors.DeadlineExceededError`
Your code must catch these exceptions, back off from calling
lease_tasks()
lease_tasks()
, and then try again later. To avoid this problem, consider setting a higher RPC
deadline when calling
lease_tasks()
lease_tasks()
. You should also back off when a lease request returns an empty list of
tasks.
If you generate more than 10 LeaseTasks requests per queue per
second, only the first 10 requests will return results. If requests exceed this
limit, OK is returned with zero results.
To view information about all the tasks and queues in your application:
-
Open the Cloud Tasks page in the Google Cloud console and look for the Pull value in column Type.
-
Click on the name of the queue in which you are interested, opening the queue details page. It displays all of the tasks in the selected queue.
Once a worker completes a task, it needs to delete the task from the queue. If you see tasks remaining in a queue after a worker finishes processing them, it is likely that the worker failed; in this case, the tasks will be processed by another worker.
You can delete an individual task or a list of tasks using
deleteTask().
You must know the name of a task in order to delete it. You can find task names
in the Task object
returned by taskqueue.Lease
lease_tasks() .
The following code sample demonstrates how to delete a task from a queue:
View TaskqueueServlet.java on GitHub (region:
delete_task)
You can delete a list of tasks, such as that returned by taskqueue.Lease, by
passing it to
taskqueue.DeleteMulti:
You can delete a list of tasks, such as that returned by taskqueue.Lease
lease_tasks() , simply by passing it to
delete_tasks():
For a simple but complete end-to-end example of using pull queues in Python, see appengine-pullqueue-counter.