Dynamic Delivery Ingest Priority Queueing

This topic explains how to use the priority setting to optimize ingestion time for high-priority videos.

Introduction

Dynamic Delivery Ingest now has a feature called Priority Queueing that allow publishers to submit ingest jobs to us with a desired priority setting to influence the order and timeliness of when the job will be processed.

How Dynamic Delivery Ingestion Queueing Works

Normal priority supports up to 200 concurrent jobs per account, after this quota is reached the system will queue the request up to 10,000. After that a new request will be rejected. The system constantly is validating concurrent jobs and is moving jobs from the queue to concurrent.

Low priority supports up to 100 concurrent jobs per account, after this quota is reached the system will queue the request up to 20,000. After that a new request will be rejected. One thing to keep on mind is low priority could take longer to be completed.

We can not ensure an ETA on low priority jobs, as there are many factors that can affect that.

The video status in the CMS API does not reflect that the job was queued - the status will be pending whether the job is processing or queued.

How Priority Queueing Affects Ingestion

Priority queueing allows the user to add a priority flag to an ingest request. The allowable values for priority are low and normal. Any other value will cause the request to be rejected with a 422 error code. When the user doesn't specify any priority, the default value of normal is used. Here is a brief description of how Priority Queueing changes how jobs are processed from the queue:

  1. If there are no queued jobs and there is capacity to run a job, then the job is run immediately. This applies to both low and normal priority jobs.
  2. If there is no capacity for another job to run, the job is queued.
  3. If there are jobs in the queue, then any new jobs are also queued. This means that a new job can't start before queued jobs.
  4. When there is the capacity to run another job, and there are queued jobs, a job is taken from the queue:
    • If there are ANY normal priority jobs in the queue, the oldest normal priority job will be picked.
    • If there are NO normal priority jobs in the queue, then the oldest low priority job will be picked.

What to watch out for

Rendition details changed after submitting a job

When you create a Dynamic Ingest job, the ingest profile specified is copied by the Dynamic Ingest system, and that profile copy will be used for processing even if the profile has been modified since the job was submitted.

For low priority jobs, where processing could be delayed for a significant amount of time in some cases, the renditions specified in that profile could be modified before the job is actually processed. If that happens, the renditions used to process the video will be the NEW definition of those renditions, not the definition of the renditions at the time the job was submitted.

If you wish to guarantee you get the renditions as they were specified when the job was submitted, you should make copies of the renditions and create a new profile that references those copied renditions, then start the ingestion process. (You can retranscode the video with the updated profile if the video has already been put into Video Cloud. Remember that you cannot retranscode the video until processing for the original job is complete.)

Source file upload

Source file upload will delete files in that S3 bucket after 24-hours. Since low priority jobs may not be processed for more than 24-hours in some cases, the file(s) may have already been deleted, and processing in that case will fail. We do not recommend low priority ingestion for source file uploads.

Sample requests

Below is a sample of an ingest request body with low priority:

{
  "master": {
    "url": "https://host/master.mp4"
  },
  "profile": "multi-platform-extended-static",
  "priority": "low",
  "callbacks": [
    "https://mydomain.com/di-callbacks.php"
  ]
}

And here's a sample of a normal priority job:

{
  "master": {
    "url": "https://host/master.mp4"
  },
  "profile": "multi-platform-extended-static",
  "priority": "normal",
  "callbacks": [
    "https://mydomain.com/di-callbacks.php"
  ]
}

Note that since normal is the default value, the previous request would be processed in exactly the same way as the following one:

{
  "master": {
    "url": "https://host/master.mp4"
  },
  "profile": "multi-platform-extended-static",
  "callbacks": [
    "https://mydomain.com/di-callbacks.php"
  ]
}

Using the priority queue

From the last example in the previous section, it should be clear that if you do not use the priority field in your ingest jobs, they will continue to be processed exactly as they have been.

The addition of the priority field will benefit you mainly if:

  1. You ingest a fairly large number of videos
  2. It is critical to get some of your videos online as soon as possible, while there is no urgency to publish others (because the content is less important, or you do not plan to publish the videos immediately anyway)
  3. You have videos that you want to retranscode, but you do not want that to slow down the ingestion of new content

How the low priority queue works

This section explains how the low priority queue works.

There is no guarantee when normal or low priority jobs will be started. However, normal priority jobs will always be started before low priority jobs.

If you have a steady, fast rate of submitting normal priority jobs, it could take considerable time before low priority jobs are started.

Low priority jobs may take longer to start and also longer to process than normal priority for the same video.