Using Live Redundancy

Introduction

Brightcove Live's redundancy feature helps to ensure reliable performance for your live events by creating a backup stream that Live will automatically failover to in the event that the first stream stops working.

Note that all requests to the Live API require the following headers:

API Request Headers
Key	Value	Notes
`X-API-KEY`	{your API Key}	Your key should have been provided when you opened your Brightcove Live account
`Content-Type`	application/json	Technically, the `content-type` header is required only for write requests that include a request body, but should do no harm on read requests.

Create your Live jobs

For the redundant setup, you will need to create 2 or more Brightcove Live Jobs. The only requirement of the Jobs is that they are created with the same output settings. The easiest way to achieve this is to create one job with the output specifications you want, and then use the copy_outputs_from_job parameter to create the additional jobs.

There is no restriction on the regions these jobs can be created in, however, it is recommended they are located relatively nearby each other.

After the Live Jobs have been created, hold onto the Job IDs for later. The request body you will use to add these jobs to a redundant group (see the sections below) will look like this:

    [
      {
        "job_id": "0b76bc73f92f46dc917bbe5061c0c633"
      },
      {
        "job_id": "0ae5a4a71dc54b3181af0f98ee407c27"
      }
    ]

Create the redundant group

To create a Redundant Group, you will submit a POST request to:

    https://api.bcovlive.io/v1/redundantgroups

Here is a sample request body:

    {
      "ad_insertion": true,
      "processing_regions": ["us-west-2"],
      "storage_regions": ["us-west-2", "us-east-1"],
      "label": "Test RG",
      "live_dvr_sliding_window_duration": 1800
    }

The table below contains a full list of fields for the request body. In several cases these are identical to fields used to create a live job. See the Live API Reference for more details on the fields.

Create Redundant Group Fields
Field	Type	Required?	Description
`ad_insertion`	boolean	optional	Set to true if this stream should be SSAI enabled
`add_cdns`	array	optional	Array of additional CDN providers to be used for manifest generation. For each CDN provided, the manifest will be prepended accordingly
`drm`	object	optional	Not yet supported
`encryption`	object	optional	Not yet supported
`label`	string	required	A label to identify the group
`live_dvr_sliding_window_duration`	integer	optional
`notifications`	array	optional	Array of notification destination objects or strings
`processing_regions`	array	required	Processing regions for the Redundant Group. This will determine which AWS regions will generate the manifests. Recommended this matches storage_regions and the regions the Live Jobs are created in.
`storage_regions`	array	required	Storage regions media chunks and playlists will be uploaded to in S3. Recommended this matches processing_regions and the regions the Live Jobs are created in.
`videocloud`	object	optional	Video Cloud customers have the option to create a video to use for the live stream.

The response will look something like this:

    {
      "id": "481ff4cf0bf74956bc2ec6e126588080",
      "processing_regions": [
          {
              "region": "us-west-2",
              "probability": 1
          }
      ],
      "storage_regions": [
          "us-west-2",
          "us-east-1"
      ],
      "jobs": [],
      "state": "standby",
      "label": "Test RG",
      "live_dvr_sliding_window_duration": 1800,
      "status": {
          "us-west-2": null
      },
      "ad_insertion": true,
      "outputs": {
          "playback_url": "https://playback-qa.a-live.io/r481ff4cf0bf74956bc2ec6e126588080/us-west-2/NA/playlist.m3u8",
          "playback_url_dvr": "https://playback-qa.a-live.io/r481ff4cf0bf74956bc2ec6e126588080/us-west-2/NA/playlist_dvr.m3u8",
          "ssai_playback_urls": {
              "26f8470f61374e608e27af9c1b3f7ff0": {
                  "playback_url": "https://playback-qa.a-live.io/r481ff4cf0bf74956bc2ec6e126588080/us-west-2/NA/26f8470f61374e608e27af9c1b3f7ff0/playlist_ssaiM.m3u8",
                  "playback_url_dvr": "https://playback-qa.a-live.io/r481ff4cf0bf74956bc2ec6e126588080/us-west-2/NA/26f8470f61374e608e27af9c1b3f7ff0/playlist_dvr_ssaiM.m3u8",
                  "description": "House Ads - 864b84f712ae40bca1510a8052b34312",
                  "type": "ads"
              }
      }
    }

Get redundant groups

You can get all redundant groups by submitting a GET request to:

https://api.bcovlive.io/v1/redundantgroups

You can filter the response by using the state param. The allowed values are:

cancelled
cancelling
deleting
disconnected
failed
finished
finishing
processing
standby
waiting

There is also a page_size parameter that can be set to an integer up to 1000. The default page_size is 10.

The response will look like this:

{
  "redundant_groups": [
    {
      "id": "91c268a6ec5240d79a6004f4ccf0dc6f",
      "account_id": "a95ac581551b4478b27910e5675db1f8",
      "user_id": "c2691d4d039040be96c190a949d754a7",
      "processing_regions": [
        {
          "region": "us-west-2",
          "probability": 1
        }
      ],
      "storage_regions": [
        "us-west-2",
        "us-east-1"
      ],
      "jobs": [],
      "state": "standby",
      "created_at": 1594316624287,
      "updated_at": 1594316624287,
      "label": "Test Redundant Group",
      "live_dvr_sliding_window_duration": 86400,
      "status": {
        "us-west-2": null
      },
      "outputs": {
        "playback_url": "https://bcovlive-a.akamaihd.net/r91c268a6ec5240d79a6004f4ccf0dc6f/us-west-2/NA/playlist.m3u8",
        "playback_url_dvr": "https://bcovlive-a.akamaihd.net/r91c268a6ec5240d79a6004f4ccf0dc6f/us-west-2/NA/playlist_dvr.m3u8"
      }
    },
    {
      "id": "279ac36e4b4d48a3abbd3e1f98cd57aa",
      "account_id": "a95ac581551b4478b27910e5675db1f8",
      "user_id": "c2691d4d039040be96c190a949d754a7",
      "processing_regions": [
        {
          "region": "us-west-2",
          "probability": 1
        }
      ],
      "storage_regions": [
        "us-west-2",
        "us-east-1"
      ],
      "jobs": [],
      "state": "standby",
      "created_at": 1594323207015,
      "updated_at": 1594323207015,
      "label": "Test Redundant Group2",
      "live_dvr_sliding_window_duration": 86400,
      "status": {
        "us-west-2": null
      },
      "outputs": {
        "playback_url": "https://bcovlive-a.akamaihd.net/r279ac36e4b4d48a3abbd3e1f98cd57aa/us-west-2/NA/playlist.m3u8",
        "playback_url_dvr": "https://bcovlive-a.akamaihd.net/r279ac36e4b4d48a3abbd3e1f98cd57aa/us-west-2/NA/playlist_dvr.m3u8"
      }
    }
  ]
}

Add live jobs to the redundant group

After you have created a redundant group, you can add jobs to it by send a POST request to:

    https://api.bcovlive.io/v1/redundantgroups/{redundant_group_id}/jobs

The live job ids are specified in the request body like this:

    [
      {
        "job_id": "0b76bc73f92f46dc917bbe5061c0c633"
      },
      {
        "job_id": "0ae5a4a71dc54b3181af0f98ee407c27"
      }
    ]

There are a couple of additional optional properties for the job objects - the following table shows all the fields:

Add Live Jobs Request Fields
Field	Type	Required?	Description
`job_id`	string	required	ID of Job to add to Group. If neither `playlist` or `streams` are specified, all `outputs` will be used.
`playlist`	string	optional	Label of playlist to use as the outputs for the stream. If `playlist` is defined, `streams` must be undefined.
`streams`	array	optional	List of stream labels to be used as outputs for the stream. If `streams` is defined, `playlist` must be undefined.

The success response for this request will just return the redundant group id.

Get redundant group status

You can get the status of a Redundant Group by submitting a GET request to:

    https://api.bcovlive.io/v1/redundantgroups/{redundant_group_id}

The response will look like the following:

    {
      "id": "481ff4cf0bf74956bc2ec6e126588080",
      "processing_regions": [
          {
              "region": "us-west-2",
              "probability": 1
          }
      ],
      "storage_regions": [
          "us-west-2",
          "us-east-1"
      ],
      "jobs": [
          {
              "job_id": "0b76bc73f92f46dc917bbe5061c0c633",
              "streams": [
                  "hls720p",
                  "hls540p",
                  "hls360p"
              ],
              "state": "processing"
          },
          {
              "job_id": "0ae5a4a71dc54b3181af0f98ee407c27",
              "streams": [
                  "hls720p",
                  "hls540p",
                  "hls360p"
              ],
              "state": "processing"
          }
      ],
      "state": "processing",
      "created_at": 1568057414849,
      "updated_at": 1568059153017,
      "label": "Test RG",
      "live_dvr_sliding_window_duration": 1800,
      "status": {
          "us-west-2": {
              "SwitchDrift": 0,
              "Ended": false,
              "on_air": "0b76bc73f92f46dc917bbe5061c0c633",
              "Mode": "auto",
              "InManifest": true,
              "MediaSequence": 10,
              "Healthiness": 0,
              "Duration": 4,
              "DiscontinuitySequence": 1,
              "SourceChunk": {
                  "MediaSequence": 3639,
                  "Duration": 4,
                  "DiscontinuitySequence": 0,
                  "ProgramDateTime": "2019-09-09T19:59:36Z",
                  "LiveJobID": "0b76bc73f92f46dc917bbe5061c0c633"
              },
              "UpdatedAt": "2019-09-09T19:59:46Z",
              "ProgramDateTime": "2019-09-09T19:59:36Z"
          }
      },
      "ad_insertion": true,
      "outputs": {
          "playback_url": "https://playback-qa.a-live.io/r481ff4cf0bf74956bc2ec6e126588080/us-west-2/NA/playlist.m3u8",
          "playback_url_dvr": "https://playback-qa.a-live.io/r481ff4cf0bf74956bc2ec6e126588080/us-west-2/NA/playlist_dvr.m3u8",
          "ssai_playback_urls": {
              "26f8470f61374e608e27af9c1b3f7ff0": {
                  "playback_url": "https://playback-qa.a-live.io/r481ff4cf0bf74956bc2ec6e126588080/us-west-2/NA/26f8470f61374e608e27af9c1b3f7ff0/playlist_ssaiM.m3u8",
                  "playback_url_dvr": "https://playback-qa.a-live.io/r481ff4cf0bf74956bc2ec6e126588080/us-west-2/NA/26f8470f61374e608e27af9c1b3f7ff0/playlist_dvr_ssaiM.m3u8",
                  "description": "House Ads - 864b84f712ae40bca1510a8052b34312",
                  "type": "ads"
              }
          }
      }
    }

Each processing region in the Redundant Group will have its own status object, keyed by region. You can see in this example that in the us_west_2 processing region, the on_air job is 0b76bc73f92f46dc917bbe5061c0c633.

As a side note, the probability associated with the processing region will at this point always be 1, but is included in the data model for future enhancements that will allow multiple processing regions with probabilities between 0 and 1, representing the approximate percentage of playback traffic that will use that region.

The Healthiness parameter is used by the Redundancy service to compare output processing times between jobs and estimate how many seconds of playback buffer should be available to players before a stall occurs. Auto failover occurs when the on_air job's Healthiness is less than or equal to 0, and when in auto mode, the service will attempt to switch to another healthy job.

Manually adding cuepoints

Manually adding cuepoints to a live stream with redundancy is very similar to adding cuepoints to an ordinary live stream. To do this for a redundant group, make a POST request to:

    https://api.bcovlive.io/v1/redundantgroups/{redundant_group_id}/cuepoint

Include a request body like the following:

    {
      "ad_server_data": {
        "subject": "wildlife"
      },
      "duration": 30,
      "timecode": "09:23:18:05"
    }

If you omit the timecode field, the cuepoint will be inserted immediately.

Force a job failover

Failover will be managed automatically by the Brightcove Live system, and should require no intervention on your part. If for some reason you want to force failover to another job, however, the simplest way to do it is to simply stop the encoder of the on_air job.

You can also force failover using the API by making a PUT request to:

    https://api.bcovlive.io/v1/redundantgroups/{redundant_group_id}/switch

The request body looks like this:

    {
      "mode": "manual",
      "on_air": "0ae5a4a71dc54b3181af0f98ee407c27"
    }

Here, mode will be either manual or auto (the default), and the on_air will have a value of one of the secondary job ids.

Note: Once in manual mode, automatic job failover will not occur. You must set the mode back to auto for automatic failover to resume. You can change the On-Air Job without switching to manual by omitting mode from the body and adding the force query param ?force=true to your request. This will force the service to switch Jobs, but it may switch back at any time if it detects issues.

Ending a redundant stream

There are two ways you can end your redundant stream. By design, Redundant Groups are implicitly SEP streams. You can put a Redundant Group into STANDBY mode by removing all Jobs from the Redundant Group.

To do this using a DELETE request to:

    https://api.bcovlive.io/v1/redundantgroups/{redundant_group_id}/jobs/{job_id}

for each job you want to remove. Note that in order to remove the job that is currently on_air, you will have to add the ?force=true query parameter.

The second way to end the stream is to delete the redundant group completely by sending a DELETE request to:

    https://api.bcovlive.io/v1/redundantgroups/{redundant_group_id}

Limitations

Live redundancy cannot be used with DRM-protected live streams.
Live jobs using the h264_profile": "baseline" in any of the outputs is not supported by Redundancy and may experience playback stalls and stability issues if used. Use "h264_profile": "main" instead