Configuring credentials for direct remote storage access

Format

When a job needs to access a distant storage server with access control, it will parse a specific configuration file to retrieve credentials. This file must be created and deployed by the user on every On-Demand Encoder server before creating any encoding jobs.

The file is in yaml format. The file is located is /etc/mediakind/mkp/charts/customer-values.yaml. A specific script must be executed to apply the changes brought to the file that remain after a product update. If no value is specified or if a value is removed from the file, default values from installer are used.

Each entry contains the following elements:

• url: the base URL of a storage.
• type: the type of credential to be used for this base URL. It can be:
  • user for access with username and password
  • s3 for access using AWS S3 object store credentials
  • azureSasToken for access using Blob Azure credential

User credential

The entry should contain the following elements:

• username
• password

It can be used for FTP or SFTP URL (ftp://... or sftp://…).

credentials:
  [ {"type": "user", "url": "sftp://myServer/share1", "username": "test", "password": "1234"} ]

Blob Azure credential

To configure the Azure authentication for your system, the entry in the credential file should contain the following elements:

• url: URL provided at the Blob storage creation.
• type: Type of credential to be used for this URL. It must be: azureSasToken.
• sasToken: Token provided at the Blob storage creation.

credentials:
    [{"type" : "azureSasToken", "url" : "https://myaccount.blob.core.windows.net/mycontainer", "sasToken" : "?sv=2020-02-10&ss=bfqt&srt=o&sp=rwdlacupx&se=2021-02-24T00:46:59Z&st=2021-02-23T16:46:59Z&spr=https&sig=..."} ]

AWS S3 object store credential

This credential corresponds to an AWS access key.

The entry should contain the following elements:

• accessKeyId: The ID of the access key
• secretAccessKey: The secret access key

credentials:
  [{ "type": "s3", "url" : "https://mk-cr8-poc.s3.us-east.cloud-object-storage.appdomain.cloud", "accessKeyId" : "e45d9e87d91421f2f3c69fca4c", "secretAccessKey":"236e8cc37c9ccc355b9ffb17fe8"} ]

Multiple credentials display

credentials:
  [ 
  {"type": "user", "url": "sftp://myServer/share1", "username": "test", "password": "1234"},
  {"type" : "azureSasToken", "url" : "https://myaccount.blob.core.windows.net/mycontainer", "sasToken" : "?sv=2020-02-10&ss=bfqt&srt=o&sp=rwdlacupx&se=2021-02-24T00:46:59Z&st=2021-02-23T16:46:59Z&spr=https&sig=..."},
  { "type": "s3", "url" : "https://mk-cr8-poc.s3.us-east.cloud-object-storage.appdomain.cloud", "accessKeyId" : "e45d9e87d91421f2f3c69fca4c", "secretAccessKey":"236e8cc37c9ccc355b9ffb17fe8"} 
  ]

Deployment

This configuration file should be created by the user and needs to be deployed on every Packager server under the following path: /etc/mediakind/mkp/charts/customer-values.yaml.

Once the customer-values.yaml file is configured, the following script must be executed to apply the changes:

# /etc/mediakind/mkp/charts/apply_customer_values.sh

Example

When a job is created, it will parse this file to find the associated credentials of the file URL it needs to access.

For example, if the file contains the following credentials:

credentials:
  [
  {"type" : "user", "url" : "sftp://server1", "username" : "admin", "password":"1234"},
  {"type" : "user", "url" : "sftp://server2/share", "username" : "admin", "password":"4321"}
  ]

If a job input file is sftp://server2/share/video.ts, the second credential will be used: the input URL matches with the base URL of the second credential.