Source File Upload API for Dynamic Ingest

Introduction

For ingest via source file upload, Brightcove provides an S3 bucket that you can upload your videos and asset files to, and Dynamic Ingest then pulls the video from S3 bucket in same way it would from your own S3 bucket or URL. The diagram below shows the difference between the workflows for basic dynamic ingest and ingest with source file upload.

FAQ

How long are videos temporarily stored, and when do the URLs for them become invalid?

Videos are removed from temporary storage after 24 hours after they are uploaded, and the URLs for them are no longer valid after that.

How long are the S3 credentials returned by the Dynamic Ingest API valid?

The S3 credentials are also valid for 24 hours after the API sends them.

Are video files physically deleted from the S3 bucket after 24 hours?

Yes

Are videos deleted from the S3 bucket after they are successfully ingested?

All videos are deleted from temporary storage after 24 hours, whether they have been successfully ingested or not.

Can the videos in temporary storage be accessed publicly by someone who has the URL?

No

Is there any way to download or view the video in temporary storage without security credentials?

No

Are the security credentials to access the temporary storage shared with other Brightcove customers?

No, any customer using the temporary storage is given unique security credentials.

Is there a way other Brightcove customers could access my videos in temporary storage using their own security credentials?

No, security credentials only provide access to the videos you pushed to temporary storage.

What region does the S3 bucket for file uploads reside it?

US-EAST-1 (this is fixed).

Source file names

To avoid issues in accessing videos and assets in the Brightcove Player, you should avoid using any special characters in source file names, whether videos, images, or text tracks (WebVTT files). This also applies to remote assets. File names should only include the following:

• Single-byte letters (upper or lower case)
• Numbers
• Dashes (-) and underscores (_)
• Spaces if they are URL-encoded

Authentication

The easiest way to get client credentials for Dynamic Ingest is through the Studio Admin page for API Authentication. For the API permissions, you need at least:

• CMS > Video Read
• Dynamic Ingest > Create
• Dynamic Ingest > Push Files (this is the new Source File Upload API)

Authentication for Brightcove API requests for push-based ingestion require one additional permission over those for other Dynamic Ingest requests:

      video-cloud/upload-urls/read

The full set of permissions needed for source file upload is:

• video-cloud/video/create
• video-cloud/video/read
• video-cloud/video/update
• video-cloud/upload-urls/read

These permissions are available in Studio. Alternatively, you can obtain client credentials to use the source file upload API directly from the OAuth API by making a POST request as follows:

Request URL

      https://oauth.brightcove.com/v4/client_credentials

Headers

• Authorization: BC_TOKEN {YOUR_BC_TOKEN}
• Content-Type: application/json

Request body

{
  "type": "credential",
  "maximum_scope": [
  {
    "identity": {
      "type": "video-cloud-account",
      "account-id": {YOUR_ACCOUNT_ID}
    },
    "operations": [
      "video-cloud/upload-urls/read",
      "video-cloud/video/create",
      "video-cloud/video/read",
      "video-cloud/video/update",
      "video-cloud/ingest-profiles/profile/write",
      "video-cloud/ingest-profiles/account/write",
      "video-cloud/ingest-profiles/profile/read",
      "video-cloud/ingest-profiles/account/read"
    ]
  }
  ],
  "name": "Source File Upload Credentials"
}

API requests

There are four API requests involved in push-based ingestion:

1. CMS API POST request to create the video object in Video Cloud (same as for pull-based ingestion)
2. Dynamic Ingest GET request to get the Brightcove S3 bucket URLs
3. PUT request to upload the source file to the Brightcove S3 bucket
4. Dynamic Ingest POST request to ingest the source file (same as for pull-based ingestion)

These requests are detailed in the sections that follow.

CMS API request

The CMS API request is the same as for any Dynamic Ingest operation to add a new video. This request is required to ingest a new video. If are replacing or adding assets to an existing video, you will not need this step - instead, you will use the existing video id in the other requests.

Request syntax

This is a POST request to:

      https://cms.api.brightcove.com/v1/accounts/{ACCOUNT_ID}/videos

Parameters

URL parameters for the request:

• {ACCOUNT_ID} - your account id

Request body

The request body consists of a JSON object containing the name (required) and other metadata for the video (optional):

{
  "name": "My Video"
}

See the API Reference for details.

Headers

The HTTP headers you need to include with the request are:

• Authorization: Bearer {ACCESS_TOKEN}
• Content-Type: application/json

Response

The response will be a JSON object containing the video metadata. The important item for the rest of the Dynamic Ingest operations is the id, which you will substitute for the {VIDEO_ID} in the requests to the Ingest API.

Request for S3 URLs

The first request to the Ingest API will retrieve the information you need to PUT your source file(s) to the Brightcove S3 bucket and then ingest from there into Video Cloud.

Request syntax

This is a GET request to:

https://ingest.api.brightcove.com/v1/accounts/{ACCOUNT_ID}/videos/{VIDEO_ID}/upload-urls/{SOURCE_NAME}

Parameters

URL parameters for the request:

• {ACCOUNT_ID} - your account id
• {VIDEO_ID} - the video id returned from CMS API request
• {SOURCE_NAME} - the video source filename - the name should not contain any URL-reserved characters such as ?, &, # or spaces

Headers

The HTTP headers you need to include with the request are:

• Authorization: Bearer {ACCESS_TOKEN}

Response

The response will be a JSON object similar to the following:

{
  "bucket": "ingestion-upload-production",
  "object_key": "57838016001/4752143002001/ed5a5ba0-1d97-4f95-a8ec-cbb786b04a37/greatblueheron.mp4",
  "access_key_id": "ACCESS_KEY_APPEARS_HERE",
  "secret_access_key": "SECRET_ACCESS_KEY_APPEARS_HERE",
  "session_token": "FQoDYXdzEKf//////////wEaDKR0wDgquq/qvkZgbyKOA7URC/9io6cmRBDkhbvxoHIKkPZlK/9YNvdWcESPkm75/2PvU6FV1Mc+/XENPzY8KgvP86MBJNxYLPdkuP1phgHs2Yh2p1KIDcQSCZJ3i6i9m4S14ewjWIugYLYDQi6CG+3fiFwfzbKT5jes1kh24m9BQQIuvVOiM1GLTldyDzlrdDopJkdYd4IEU7FU36CUT7RL/aeMwR2Usk56nwqyqkkQHPmvqmGyiLdrD3OrIbUU+6+ZP4usS9dbV3eAqOWDIk3HCN+Kuc9f/eUWhY21ftNDXWgasqQqXwPRs3T1i/hoiIKODbzr8F",
  "signed_url": "https://ingestion-upload-production.s3.amazonaws.com/57838016001/4752143002001/ed5a5ba0-1d97-4f95-a8ec-cbb786b04a37/greatblueheron.mp4?AWSAccessKeyId=ACCESS_KEY_HERE&Expires=1475673952&Signature=%2Fsr5cV%2FVOfGCBkodol9xQIKlbu4%3D",
  "api_request_url": "https://ingestion-upload-production.s3.amazonaws.com/57838016001/4752143002001/ed5a5ba0-1d97-4f95-a8ec-cbb786b04a37/greatblueheron.mp4"
}

The items in the response are:

• bucket - the S3 bucket name
• object_key - the object key for the file upload (used in constructing the destination URL for multipart uploads)
• access_key_id - the access key used for authenticating the upload request (used for multipart uploads)
• secret_access_key - the secret access key used for authenticating the upload request (used for multipart uploads)
• session_token - a short-lived AWS token that provides the capability to write to the target objec
• signed_url - this is a shorthand S3 url that you can PUT your source file(s) to if you have relatively small videos and are not implementing multipart upload
• api_request_url - this is the URL you will include in your Dynamic Ingest POST request for the Master url or url for the image/text_tracks assets

It is recommended that you use multipart upload using the AWS SDK for the language you are using. SDKs are available for many languages, including Java, .NET, Ruby, PHP, Python, JavaScript, Go, and C++. See the AWS Developer Blog for more information.

If you are implementing multipart upload, the following documents and sample code will be useful:

• Upload a File
• Java sample
• Python sample

Here is simple example in PHP:

<?php
  // AWS SDK (for push ingests)
  require 'vendor/aws-autoloader.php';
  
  use Aws\S3\S3Client;
  use Aws\S3\MultipartUploader;
  use Aws\Exception\MultipartUploadException;
  
  /**
    * get S3 information as described above in this doc
    * the code below assumes it has been decoded as $s3response
    * and that $filePath is the local path to the asset file
    */
  
  s3 = new S3Client([
      'version' => 'latest',
      'region'  => 'us-east-1',
      'credentials' => array(
          'key'    => $s3response->access_key_id,
          'secret' => $s3response->secret_access_key,
          'token'	 => $s3response->session_token
      )
  ]);
  $params = array(
      'bucket' => $s3response->s3->bucket,
      'key' => $s3response->s3->object_key
  );
  $uploader = new MultipartUploader($this->s3, $filePath, $params);
  try {
      $uploadResponse = $uploader->upload();
  } catch (MultipartUploadException $e) {
      echo $e->getMessage() . "\n";
  }
?>

PUT source file(s) to S3

After getting the S3 URLs, you make a PUT request to upload your video file, using the signed_url as the destination.

You can use the following curl command to test the PUT operation:

      curl -X PUT "SIGNED_URL_GOES_HERE" --upload-file FILE_PATH_FOR_LOCAL_ASSET_GOES_HERE 

If you are using Postman to send the file to S3, you will need to uncheck the Content-Type header:

Also be sure to change the body type to Binary and select your video file for upload:

Single vs. multipart upload

AWS allows single-part uploads for files up to 5 GB in size (there are no other limits on file size). For larger files, you must use multipart uploads. Although single-part upload is somewhat easier to set up, we recommend using multipart uploading whenever possible. Here are the differences between the two:

• Single part upload uploads the video all as one file. Single part upload is limited to file sizes of 5 GB or less. If the upload is interrupted for some reason, you must start over.
• Multipart upload pushes the file in chunks. This is more efficient because the upload can take advantage of multiple connections. Also, if the upload is interrupted, it can resume where it left off with the remaining chunks.

Dynamic Ingest request

After your file has uploaded to the Brightcove S3 bucket, you make an ordinary Dynamic Ingest request to ingest the file from its S3 location.

Request syntax

This is a POST request to:

      https://ingest.api.brightcove.com/v1/accounts/{ACCOUNT_ID}/videos/{VIDEO_ID}/ingest-requests

Parameters

URL parameters for the request:

• {ACCOUNT_ID} - your account id
• {VIDEO_ID} - the video id returned from CMS API request

Request body

The request body consists of a JSON object containing the master (required) details for the ingest job. The url for the master will be the api_request_url returned by the request for the S3 bucket information

      {
      "master": {
          "url": "https://ingestion-upload-prod.s3.amazonaws.com/12345/5678/3712cd37504911ab06a77a26a387ce/source.mp4"
      },
      "profile": "multi-platform-standard-static",
      "capture-images": true
      }

See the API Reference for details.

Headers

The HTTP headers you need to include with the request are:

• Authorization: Bearer {ACCESS_TOKEN}
• Content-Type: application/json

Response

The response will contain the job_id for the ingest request, which allows you to track the status via Notifications.

Sample code

To help you get started with push-based Dynamic Ingest, we have created some sample apps in Java and Python. You can find them on our Github site.