Get Status of Dynamic Ingest Requests

When you use the Dynamic Ingest API to add videos to your Video Cloud account, what you most want to know is when the video has been processed and whether or not renditions were successfully created.

Introduction

This document explains how you can track the status of jobs by either making status requests to the CMS API or by using Dynamic Ingest API notifications. We also provide a sample dashboard app that automates the process

Note that the status of ingest jobs is available only for jobs submitted with the past 7 days.

Requesting status

You get the status of dynamic ingest jobs (ingest, replace, or retranscode) using these CMS API endpoints - note that these endpoints works for Dynamic Delivery jobs only:

Get status for all jobs

      https://cms.api.brightcove.com/v1/accounts/{{account_id}}/videos/{{video_id}}/ingest_jobs
    

The response will look something like this:

      [
        {
        "id": "ac49b1db-e6e1-477f-a2c1-70b9cd3107cb",
        "state": "finished",
        "account_id": "57838016001",
        "video_id": "5636411346001",
        "error_code": null,
        "error_message": null,
        "updated_at": "2017-11-07T13:56:51.505Z",
        "started_at": "2017-11-07T13:56:12.510Z",
        "priority": "normal",
        "submitted_at": "2017-11-07T13:56:12.435Z"
        },
        {
        "id": "10605652-8b6f-4f22-b190-01bd1938677b",
        "state": "processing",
        "account_id": "57838016001",
        "video_id": "5636411346001",
        "error_code": null,
        "error_message": null,
        "updated_at": null,
        "started_at": null,
        "priority": "low",
        "submitted_at": "2017-11-07T14:06:35.000Z"
        }
        ]
    

Get status for a specific jobs

      https://cms.api.brightcove.com/v1/accounts/{{account_id}}/videos/{{video_id}}/ingest_jobs/{job_id}
    

The response will look something like this:

      {
        "id": "ac49b1db-e6e1-477f-a2c1-70b9cd3107cb",
        "state": "finished",
        "account_id": "57838016001",
        "video_id": "5636411346001",
        "error_code": null,
        "error_message": null,
        "updated_at": "2017-11-07T13:56:51.505Z",
        "started_at": "2017-11-07T13:56:12.510Z",
        "priority": "normal",
        "submitted_at": "2017-11-07T13:56:12.435Z"
        }
    

The possible values for state are:

  • processing: processing, video is not playable yet
  • publishing: at least one playable rendition has been created, and the video is being readied for playback
  • published: at least one rendition is a available for playback
  • finished: at least one audio/video rendition has been processed
  • failed: processing failed; if you are not able to figure out what went wrong, contact Support

Getting notifications

While the request status method discussed above works, if you are waiting for a particular state (published or finished), it is better to let Brightcove notify you when these events occur rather than have to keep asking for the status until you get the answer you are looking for. We will now look at how you might build an app around handling notifications.

The Dynamic Ingest notifications give you all the information you need to know when your video is ready - you just need to know what to look for...and to define what "ready" means for your systems. This diagram summarizes the workflow:

Ingest Status Workflow
Ingest Status Workflow

Dynamic Ingest Notifications

The Dynamic Ingest notification service sends you notifications for several kinds of events. The two that are most useful for figuring out when the video is "ready" are ones indicating that particular renditions have been created and the one that indicates that all processing is complete. Here are examples of each:

Dynamic rendition created notification

        {
          "entity": "default/video3800",
          "entityType": "DYNAMIC_RENDITION",
          "version": "1",
          "action": "CREATE",
          "jobId": "d3ef8751-2b88-4141-95d5-83f0393aca07",
          "videoId": "5660367449001",
          "dynamicRenditionId": "default\/video3800",
          "bitrate": 3804,
          "width": 1920,
          "height": 1080,
          "accountId": "57838016001",
          "status": "SUCCESS"
          }
        
      
Notification for Rendition Created

Note in this example:

  • The videoId value lets you know which video the rendition is for (in case you have multiple ingest jobs running)
  • The entity value is the dynamic rendition type created
  • if the status value is "SUCCESS", the rendition was created successfully

Processing complete notification

        {
          "entity": "5660367449001",
          "entityType": "TITLE",
          "version": "1",
          "action": "CREATE",
          "jobId": "d3ef8751-2b88-4141-95d5-83f0393aca07",
          "videoId": "5660367449001",
          "accountId": "57838016001",
          "status": "SUCCESS"
          }
        
      
Notification for Processing Complete

Note in this example:

  • The videoId and jobId values let you know which video this is for (in case you have multiple ingest jobs running)
  • If the status value is "SUCCESS", the video was processed successfully

To receive notifications, you need to include a "callbacks" field in you Dynamic Ingest API requests, pointing to one or more callback addresses:

      {
        "master": {
        "url": "https://s3.amazonaws.com/bucket/mysourcevideo.mp4"
        }, "profile": "multi-platform-extended-static",
        "callbacks": ["https://host1/path1”, “https://host2/path2”]
        }
      
    

Sample Dashboard

This section explains how notifications can be put together to build a simple dashboard for the Dynamic Ingest API. The handler for notifications parses notifications from the Dynamic Ingest API to identify processing complete notifications. It then adds the video notifications into an array of objects for each video in a JSON file. The dashboard itself is an HTML page that imports the JSON file to get the notification data. It uses the ids to makes a request to the CMS API to get the video metadata.

Here is the high-level architecture of the app:

Ingest Dashboard Architecture
Ingest Dashboard Architecture

The app parts

The handler for notifications is built in PHP - it looks for processing complete notifications and adds the video id to an array in a separate JavaScript file:

      <?php
        // POST won't work for JSON data
        $problem = "No errors";
        try {
        $json = file_get_contents('php://input');
        $decoded = json_decode($json, true);
        } catch (Exception $e) {
        $problem = $e->getMessage();
        echo $problem;
        }

        // full notification
        $notification = json_encode($decoded, JSON_PRETTY_PRINT);

        // Begin by extracting the useful parts of the notification
        // for Dynamic Delivery, look for 'videoId'

        if (isset($decoded["videoId"])) {
        $videoId = $decoded["videoId"];
        } elseif (isset($decoded["entity"])) {
        $videoId = $decoded["entity"];
        } else {
        $videoId = null;
        }

        if (isset($decoded["entityType"])) {
        $entityType = $decoded["entityType"];
        } else {
        $entityType = null;
        }

        if (isset($decoded["status"])) {
        $status = $decoded["status"];
        } else {
        $status = null;
        }

        if (isset($decoded["action"])) {
        $action = $decoded["action"];
        } else {
        $action = null;
        }

        // if notification is for completed title, act

        if (($entityType == 'TITLE') && ($action == 'CREATE')) {
        if (($status == 'SUCCESS') || ($status == 'FAILED')) {
        $newLine = "\nvideoIdArray.unshift(".$videoId.");";
        // Tell PHP where it can find the log file and tell PHP to open it
        // and add the string we created earlier to it.
        $logFileLocation = "video-ids.js";
        $fileHandle = fopen($logFileLocation, 'a') or die("-1");
        chmod($logFileLocation, 0777);
        fwrite($fileHandle, $newLine);
        fclose($fileHandle);
        }
        }

        // save full notification for audit trail
        $logEntry = $notification.",\n";

        $logFileLocation = "full-log.txt";
        $fileHandle = fopen($logFileLocation, 'a') or die("-1");
        chmod($logFileLocation, 0777);
        fwrite($fileHandle, $logEntry);
        fclose($fileHandle);


        echo "Dynamic Ingest callback app is running";
        ?>
      
    

JSON file:

The JSON file is initially an empty array ( [] ) - data is added by the notification handler.

Dashboard

The dashboard includes the HTML and JavaScript to fetch the notification data and additional video data from the CMS API and write the results into a table:

      <!DOCTYPE html>
        <html>
        <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
        <title>Dynamic Ingest Log</title>
        <style>
        body {
        font-family: sans-serif;
        margin: 5em;
        }
        .hide {
        display: none;
        }
        .show {
        display: block;
        }
        table {
        border-collapse: collapse;
        border: 1px #999999 solid;
        }
        th {
        background-color: #666666;
        color: #f5f5f5;
        padding: .5em;
        font-size: .7em;
        }
        td {
        border: 1px #999999 solid;
        font-size: .7em;
        padding: .5em
        }
        .hidden {
        display: none;
        }
        </style>
        </head>
        <body>
        <h1>Dynamic Ingest Log</h1>
        <h2>Account: Brightcove Learning (57838016001)</h2>
        <p style="width:70%">
        Videos are listed in order of processing completion time, newest to oldest. The reference id (generated by the
        <a href="./di-tester.html">Dynamic Ingest tester</a>) is a combination of the date/time that the
        Dynamic Ingest job was initiated and the ingest profile that was used. You can add additional videos using the
        <a href="./di-tester.html">Dynamic Ingest tester</a>. New videos will appear in this log after
        processing is complete.
        </p>
        <p>
        <button id="clearLogBtn">Clear the log</button>
        </p>
        <div id="videoLogBlock">
        <table>
        <thead>
        <tr>
        <th>Video ID</th>
        <th>Name</th>
        <th>Reference ID</th>
        <th>Renditions Created</th>
        <th>Processing Complete</th>
        </tr>
        </thead>
        <tbody id="logBody"></tbody>
        </table>
        <h4 id="loadingMessage">Loading data, please wait...</h4>
        </div>
        <script>
        var BCLS = ( function (window, document) {
        // to use another account, set the account_id value appropriately
        // the client_id and client_secret will also need to be changed in the proxy
        var my_account_id = 57838016001,
        account_id = my_account_id,
        logBody = document.getElementById('logBody'),
        loadingMessage = document.getElementById('loadingMessage'),
        clearLogBtn = document.getElementById('clearLogBtn'),
        i = 0,
        iMax,
        // set the proxyURL to the location of the proxy app that makes Brightcove API requests
        proxyURL = './brightcove-learning-proxy.php',
        dataFileURL = './di.json',
        videoDataArray = [],
        requestOptions = {},
        currentVideo,
        currentIndex = 0;
        /**
        * tests for all the ways a variable might be undefined or not have a value
        * @param {*} x the variable to test
        * @return {Boolean} true if variable is defined and has a value
        */
        function isDefined(x) {
        if ( x === '' || x === null || x === undefined || x === NaN) {
        return false;
        }
        return true;
        }
        /**
        * find index of an object in array of objects
        * based on some property value
        *
        * @param {array} targetArray - array to search
        * @param {string} objProperty - object property to search
        * @param {string|number} value - value of the property to search for
        * @return {integer} index of first instance if found, otherwise returns null
        */
        function findObjectInArray(targetArray, objProperty, value) {
        var i, totalItems = targetArray.length, objFound = false;
        for (i = 0; i < totalItems; i++) {
        if (targetArray[i][objProperty] === value) {
        objFound = true;
        return i;
        }
        }
        if (objFound === false) {
        return null;
        }
        }
        /**
        * factory for new video objects
        * @param {String} videoId the video id
        * @return {object} the new object
        */
        function makeVideoDataObject(videoId) {
        var obj = {};
        obj.id = videoId;
        obj.name = '';
        obj.reference_id = '';
        obj.renditions = 0;
        obj.complete = 'no';
        return obj;
        }
        /**
        * processes notification objects
        * creates a new object in the videoDataArray if it doesn't exist
        * and updates the videoDataArray object based on the notification
        * @param {Object} notificationObj the raw notification object
        */
        function processNotification(notificationObj) {
        var objIndex, videoObj;
        // if notification object contains a video id, find the corresponding
        // object in the videoDataArray or create it if it's not there
        if (isDefined(notificationObj) && isDefined(notificationObj.videoId)) {
        objIndex = findObjectInArray(videoDataArray, 'id', notificationObj.videoId);
        // if not found, create one
        if (!isDefined(objIndex)) {
        videoObj = makeVideoDataObject(notificationObj.videoId);
        videoDataArray.push(videoObj);
        objIndex = videoDataArray.length - 1;
        }
        // now update properties based on what's in the notification
        if (notificationObj.entityType === 'DYNAMIC_RENDITION') {
        // increment the renditions account
        videoDataArray[objIndex].renditions++;
        }
        } else if (notificationObj.entityType === 'TITLE') {
        // overall processing notification - checked for SUCCESS / FAILED
        if (notificationObj.status === 'SUCCESS') {
        // mark complete
        videoDataArray[objIndex].complete = 'yes';
        } else if (notificationObj.status === 'FAILED') {
        // mark failed
        videoDataArray[objIndex].complete = 'failed';
        }
        }
        return;
        }
        /**
        * creates the dashboard table body
        */
        function writeReport() {
        var j,
        jMax = videoDataArray.length,
        item,
        t;
        loadingMessage.textContent = 'This page will refresh in 1 minute...';
        for (j = 0; j < jMax; j++) {
        item = videoDataArray[j];
        if (item.id !== undefined) {
        logBody.innerHTML += '<tr><td>' + item.id + '</td><td>' + item.name +
        '</td><td>' + item.reference_id + '</td><td>' + item.renditions +
        '</td><td>' + item.complete + '</td></tr>';
        }
        }
        // set timeout for refresh
        t = window.setTimeout(init, 60000);
        };
        // function to set up the notification data request
        function setJSONRequestOptions() {
        submitRequest(null, dataFileURL, 'notificationData');
        }
        // function to set up video data request
        function setVideoRequestOptions() {
        requestOptions = {};
        requestOptions.url = 'https://cms.api.brightcove.com/v1/accounts/' + account_id + '/videos/' + currentVideo.id;
        submitRequest(requestOptions, proxyURL, 'video');
        }
        /**
        * initiates the CMS API requests
        */
        function getVideoInfo() {
        iMax = videoDataArray.length;
        if (currentIndex < iMax) {
        currentVideo = videoDataArray[currentIndex];
        setVideoRequestOptions();
        } else {
        loadingMessage.innerHTML = 'No videos have been ingested - you can add some using the <a
        href="./di-tester.html">Dynamic Ingest tester</a>';
        }
        }
        /**
        * make the CMS API requests
        * @param {Object} options request options
        * @param (String) url URL to send request to
        * @param (String) type the request type
        */
        function submitRequest(options, url, type) {
        var httpRequest = new XMLHttpRequest(),
        requestData,
        responseData,
        videoDataObject,
        parsedData,
        getResponse = function () {
        try {
        if (httpRequest.readyState === 4) {
        if (httpRequest.status === 200) {
        responseData = httpRequest.responseText;
        switch (type) {
        case 'notificationData':
        var k, kMax, dataArray;
        dataArray = JSON.parse(responseData);
        // process the notifications
        kMax = dataArray.length;
        for (k = 0; k < kMax; k++) {
        processNotification(dataArray[k]);
        }
        getVideoInfo();
        break;
        case 'video':
        parsedData = JSON.parse(responseData);
        videoDataArray[currentIndex].reference_id = parsedData.reference_id;
        videoDataArray[currentIndex].name = parsedData.name;
        currentIndex++;
        if (currentIndex < iMax) {
        currentVideo = videoDataArray[currentIndex];
        setVideoRequestOptions();
        } else {
        writeReport();
        }
        break;
        }
        } else {
        console.log('There was a problem with the request. Request returned '', httpRequest.status);
        if (type === 'video') {
        setVideoRequestOptions();
        } else {
        setSourcesRequestOptions();
        }
        }
        }
        }
        catch(e) {
        console.log('Caught Exception: ', e);
        }
        };
        // notifications data is a special case
        if (type === 'notificationData') {
        // set response handler
        httpRequest.onreadystatechange = getResponse;
        // open the request
        httpRequest.open("GET", url);
        // set headers
        httpRequest.setRequestHeader("Content-Type", "application/json");
        // open and send request
        httpRequest.send();
        } else {
        // requests via proxy
        // set up request data
        requestData = "url=" + encodeURIComponent(options.url) + "&requestType=GET";
        // set response handler
        httpRequest.onreadystatechange = getResponse;
        // open the request
        httpRequest.open("POST", url);
        // set headers
        httpRequest.setRequestHeader("Content-Type", "application/x-www-form-urlencoded");
        // open and send request
        httpRequest.send(requestData);
        }
        };
        // event handlers
        clearLogBtn.addEventListener('click', function () {
        if (window.confirm('Are you sure? This action cannot be undone!')) {
        // if your clear-log app resides in another location, change the URL
        window.location.href = 'clear-log.php';
        }
        });
        // get things started
        function init() {
        // clear table and the video data array
        logBody.innerHTML = "";
        videoDataArray = [];
        setJSONRequestOptions();
        }
        // kick off the app
        init();
        })(window, document);
        </script>
        </body>
        </html>
      
    

Proxy

      <?php
        /**
        * brightcove-learning-proxy.php - proxy for Brightcove RESTful APIs
        * gets an access token, makes the request, and returns the response
        * Accessing:
        * URL: https://solutions.brightcove.com/bcls/bcls-proxy/bcsl-proxy.php
        * (note you should *always* access the proxy via HTTPS)
        * Method: POST
        *
        * @post {string} url - the URL for the API request
        * @post {string} [requestType=GET] - HTTP method for the request
        * @post {string} [requestBody=null] - JSON data to be sent with write requests
        *
        * @returns {string} $response - JSON response received from the API
        */
        // CORS entablement
        header("Access-Control-Allow-Origin: *");
        // set up request for access token
        $data = array();
        //
        // change the values below to use this proxy with a different account
        //
        $client_id = "YOUR_CLIENT_ID_HERE";
        $client_secret = "YOUR_CLIENT_SECRET_HERE";
        $auth_string = "{$client_id}:{$client_secret}";
        $request = "https://oauth.brightcove.com/v4/access_token?grant_type=client_credentials";
        $ch = curl_init($request);
        curl_setopt_array($ch, array(
        CURLOPT_POST => TRUE,
        CURLOPT_RETURNTRANSFER => TRUE,
        CURLOPT_SSL_VERIFYPEER => FALSE,
        CURLOPT_USERPWD => $auth_string,
        CURLOPT_HTTPHEADER => array(
        'Content-type: application/x-www-form-urlencoded',
        ),
        CURLOPT_POSTFIELDS => $data
        ));
        $response = curl_exec($ch);
        curl_close($ch);
        // Check for errors
        if ($response === FALSE) {
        die(curl_error($ch));
        }
        // Decode the response
        $responseData = json_decode($response, TRUE);
        $access_token = $responseData["access_token"];
        // set up the API call
        // get data
        if ($_POST["requestBody"]) {
        $data = json_decode($_POST["requestBody"]);
        } else {
        $data = array();
        }
        // get request type or default to GET
        if ($_POST["requestType"]) {
        $method = $_POST["requestType"];
        } else {
        $method = "GET";
        }
        // get the URL and authorization info from the form data
        $request = $_POST["url"];
        //send the http request
        $ch = curl_init($request);
        curl_setopt_array($ch, array(
        CURLOPT_CUSTOMREQUEST => $method,
        CURLOPT_RETURNTRANSFER => TRUE,
        CURLOPT_SSL_VERIFYPEER => FALSE,
        CURLOPT_HTTPHEADER => array(
        'Content-type: application/json',
        "Authorization: Bearer {$access_token}",
        ),
        CURLOPT_POSTFIELDS => json_encode($data)
        ));
        $response = curl_exec($ch);
        curl_close($ch);
        // Check for errors
        if ($response === FALSE) {
        echo "Error: "+$response;
        die(curl_error($ch));
        }
        // Decode the response
        // $responseData = json_decode($response, TRUE);
        // return the response to the AJAX caller
        echo $response;
        ?>
      
    

Clear the log

This simple PHP app just restores the JavaScript file to its original state, clearing out the old video ids:

      <?php
        $logFileLocation = "di.json";
        $freshContent = array ();
        $encodedContent = json_encode($freshContent);
        file_put_contents($logFileLocation, $encodedContent);
        echo 'Log file cleared - <a href="di-log.html">go back to the dashboard</a>';
        ?>