iconik Storage Gateway

Hybrid-Cloud Storage Gateway

iconik Storage Gateway Introduction

The iconik Storage Gateway (ISG) allows you to use your on-premise storage with iconik in the cloud.

Sync your folder structures from your local SAN, NAS or personal computer to the cloud, using fast parallel original file uploads, or using local transcoding to just upload proxies and keyframes.

The iconik Storage Gateway is compatible with:

  • Windows 10, Windows Server 2012 or Windows Server 2016
  • Mac OS
  • Linux: CentOS 6/7 or Ubuntu 16.04

The installation can utilize a locally installed FFMpeg or Telestream Vantage for local transcoding on all supported platforms.

Setting up

This section describes how to set up the iconik Storage Gateway. Please refer to Installing below for you specific platform information

Installing ISG

The first step is to install the ISG. Please refer to the Installation section below for your operating system.

There you will also learn where to find the configuration file that is referred to in this section.

Creating a storage

The next step is to create a storage in iconik representing the local storage this ISG will monitor.

In the Admin section, under storages, create a new storage:

Storage Purpose
FILES
Storage Type
File

Make sure to give it a name you understand. On the next page set up the configuration of the storage

Set the paths to scan, select if you want files to be automatically uploaded, and whether to use local transcoding, and which Group should be given access to the assets that get created

Make sure you have at least Read and Scan permissions set on the storage.

If you want to ignore some files/folders in your mounted point, add ignore patterns to scan_ignore field. Examples:

*exclude* (match file/folder with substring `exclude`) *.pdf (match file/folder that ends on `.pdf`)

You can also use regular expressions:

re:/^.*\.png$/ (match file/folder with `.png` ending)

When you are done setting everything up, you have to press Edit to be able to add local transcoders, and note the storage id.

If you want local transcoding, make sure you add the local transcoders in the transcoder pane to the right.

Note the storage_id, and put that in to the configuration file. You find this in the URL when you are editing the storage in the format 6f9c5e54-fb9f-11e7-8152-0a580a000240

Creating an application Token

The third step is to create an application and a token that the ISG will use for authenticating to Iconik.

In the Admin section, under Settings / Application Tokens, create a new application and select the user that the ISG will act as.

Copy the App ID and Token to the corresponding places in the configuration file.

Starting ISG

Now you are ready to start the ISG. Again refer to the Installation section below for your operating system.

Installing and operating

This section describes how to install ISG on the different platforms

CentOS 7

Set up the Cantemo Yum repository, and install the Iconik Storage Gateway RPM

cd /etc/yum.repos.d sudo wget http://yum.cantemo.com/7/cantemo-prod.repo sudo yum install iconik_storage_gateway

It will be installed in /opt/cantemo/iconik_storage_gateway where you will also find the config file called config.ini

Edit config.ini to look like this (See Setting up above how to get the correct values)

[main] app-id = 22f34c6e-268e-11e7-ac53-6c4008b85488 storage-id = 15fa3f68-7693-11e7-bf0c-02420a021013 iconik-url = https://app.iconik.io/ auth-token = eyJhbGciOiJIUzI1NiIsImlhdCI6MTUwMTU3NjAyNCwiZXhwIjoxNTAxNjE5MjI0fQ.eyJpZCI6IjMzMjFkMjIyLTc2OTMtMTFlNy1hN2EyLTAyNDIwYTAyMTAxYSJ9.sgag4OQb19J9qT1IAdKZYAqbpIoai7oW60Mahye7o8s sleep-time = 10 max-transcoding-jobs = 1 log-filename = /var/log/cantemo/iconik_storage_gateway/default.log

Iconik Storage Gateway will add a system user called cantemo that will be used to run systemd service. The cantemo user must have permissions to read/write the storage mount point folder and it contents. You can add the cantemo user to a group to manage permissions on group level instead.

After installation you can start service with:

sudo systemctl enable iconik_storage_gateway sudo systemctl start iconik_storage_gateway

To check status of the service use:

sudo systemctl status iconik_storage_gateway

For following the log file, you can use the following command:

sudo tail -f /var/log/cantemo/iconik_storage_gateway/default.log

You can run also run the systemd service from your custom user providing it to /usr/lib/systemd/system/iconik_storage_gateway.service with passing your own config and log files that are accessible for your custom user. Log file location can also be set in config.

[Service] User=john Group=john ExecStart=/usr/bin/iconik_storage_gateway --config=/home/john/iconik_storage_gateway/config.ini --log-filename=/home/john/iconik_storage_gateway/default.log

After adding your custom user you need to restart service.

sudo systemctl daemon-reload sudo systemctl restart iconik_storage_gateway

Local transcoding (Optional)

For local transcoding to work you will have to install ffmpeg and ImageMagick. Make sure they are in the path for the user running the service.

Below is one way of doing so, but please do not use on a production setup without checking compatibility with other software:

sudo yum install epel-release -y sudo yum update -y sudo rpm --import http://li.nux.ro/download/nux/RPM-GPG-KEY-nux.ro sudo rpm -Uvh http://li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-5.el7.nux.noarch.rpm sudo yum install ffmpeg ffmpeg-devel ImageMagick -y

Update the storage settings in Iconik to say `Local keyframes` and `Local proxy` and restart the service.

CentOS 6

Set up the Cantemo Yum repository, and install the Iconik Storage Gateway RPM

cd /etc/yum.repos.d sudo wget http://yum.cantemo.com/6/cantemo-prod.repo sudo yum install iconik_storage_gateway

It will be installed in /opt/cantemo/iconik_storage_gateway where you will also find the config file called config.ini

Edit config.ini to look like this (See Setting up above how to get the correct values)

[main] app-id = 22f34c6e-268e-11e7-ac53-6c4008b85488 storage-id = 15fa3f68-7693-11e7-bf0c-02420a021013 iconik-url = https://app.iconik.io/ auth-token = eyJhbGciOiJIUzI1NiIsImlhdCI6MTUwMTU3NjAyNCwiZXhwIjoxNTAxNjE5MjI0fQ.eyJpZCI6IjMzMjFkMjIyLTc2OTMtMTFlNy1hN2EyLTAyNDIwYTAyMTAxYSJ9.sgag4OQb19J9qT1IAdKZYAqbpIoai7oW60Mahye7o8s sleep-time = 10 max-transcoding-jobs = 1 log-filename = /var/log/cantemo/iconik_storage_gateway/default.log

Iconik Storage Gateway will add a system user called cantemo that will be used to run systemd service. The cantemo user must have permissions to read/write the storage mount point folder and it contents. You can add the cantemo user to a group to manage permissions on group level instead.

After installation you can start service with:

sudo /etc/init.d/iconik_storage_gateway start

For following the log file, you can use the following command:

sudo tail -f /var/log/cantemo/iconik_storage_gateway/default.log

Local transcoding (Optional)

For local transcoding to work you will have to install ffmpeg and ImageMagick. Make sure they are in the path for the user running the service.

Below is one way of doing so, but please do not use on a production setup without checking compatibility with other software:

sudo yum install epel-release -y sudo yum update -y sudo rpm --import http://li.nux.ro/download/nux/RPM-GPG-KEY-nux.ro sudo rpm -Uvh http://li.nux.ro/download/nux/dextop/el6/x86_64/nux-dextop-release-0-2.el6.nux.noarch.rpm sudo yum install ffmpeg ffmpeg-devel ImageMagick -y

Update the storage settings in Iconik to say `Local keyframes` and `Local proxy` and restart the service.

Ubuntu 16.04

Set up the Cantemo APT repository, and install the Iconik Storage Gateway package

sudo apt-get install wget sudo wget -O - http://apt.cantemo.com/ubuntu/dists/xenial/deb-public-key.asc|apt-key add - sudo echo "deb http://apt.cantemo.com/ubuntu xenial main" > /etc/apt/sources.list.d/cantemo.list sudo apt-get update sudo apt-get install iconik-storage-gateway

It will be installed in /opt/cantemo/iconik_storage_gateway where you will also find the config file called config.ini

Edit config.ini to look like this (See Setting up above how to get the correct values)

[main] app-id = 22f34c6e-268e-11e7-ac53-6c4008b85488 storage-id = 15fa3f68-7693-11e7-bf0c-02420a021013 iconik-url = https://app.iconik.io/ auth-token = eyJhbGciOiJIUzI1NiIsImlhdCI6MTUwMTU3NjAyNCwiZXhwIjoxNTAxNjE5MjI0fQ.eyJpZCI6IjMzMjFkMjIyLTc2OTMtMTFlNy1hN2EyLTAyNDIwYTAyMTAxYSJ9.sgag4OQb19J9qT1IAdKZYAqbpIoai7oW60Mahye7o8s sleep-time = 10 max-transcoding-jobs = 1 log-filename = /var/log/cantemo/iconik_storage_gateway/default.log

Iconik Storage Gateway will add a system user called cantemo that will be used to run systemd service. The cantemo user must have permissions to read/write the storage mount point folder and it contents. You can add the cantemo user to a group to manage permissions on group level instead.

After installation you can start service with:

sudo systemctl enable iconik_storage_gateway sudo systemctl start iconik_storage_gateway

To check status of the service use:

sudo systemctl status iconik_storage_gateway

For following the log file, you can use the following command:

sudo tail -f /var/log/cantemo/iconik_storage_gateway/default.log

You can run also run the systemd service from your custom user providing it to /usr/lib/systemd/system/iconik_storage_gateway.service with passing your own config and log files that are accessible for your custom user. Log file location can also be set in config.

[Service] User=john Group=john ExecStart=/usr/bin/iconik_storage_gateway --config=/home/john/iconik_storage_gateway/config.ini --log-filename=/home/john/iconik_storage_gateway/default.log

After adding your custom user you need to restart service.

sudo systemctl daemon-reload sudo systemctl restart iconik_storage_gateway

Local transcoding (Optional)

For local transcoding to work you will have to install ffmpeg and ImageMagick. Make sure they are in the path for the user running the service.

Below is one way of doing so, but please do not use on a production setup without checking compatibility with other software:

sudo apt-get install software-properties-common sudo add-apt-repository ppa:jonathonf/ffmpeg-3 sudo apt-get install ffmpeg imagemagick

Update the storage settings in Iconik to say `Local keyframes` and `Local proxy` and restart the service.

Windows

Download and install the Iconik Storage Gateway

https://iconik.io/downloads/IconikStorageGateway-latest.msi

It will be installed in \Program Files\Cantemo\Iconik Storage Gateway where you will also find the config file called config.ini

Edit config.ini to look like this (See Setting up above how to get the correct values)

[main] app-id = 22f34c6e-268e-11e7-ac53-6c4008b85488 storage-id = 15fa3f68-7693-11e7-bf0c-02420a021013 iconik-url = https://app.iconik.io/ auth-token = eyJhbGciOiJIUzI1NiIsImlhdCI6MTUwMTU3NjAyNCwiZXhwIjoxNTAxNjE5MjI0fQ.eyJpZCI6IjMzMjFkMjIyLTc2OTMtMTFlNy1hN2EyLTAyNDIwYTAyMTAxYSJ9.sgag4OQb19J9qT1IAdKZYAqbpIoai7oW60Mahye7o8s sleep-time = 10 max-transcoding-jobs = 1 log-filename = C:\Users\XYZ\iconik_storage_gateway.log

Iconik Storage Gateway will be installed as a system Service.

Start the service, and make sure it is set to Automatic if you want it to start at boot.

Local transcoding (Optional)

For local transcoding to work you will have to install ffmpeg and ImageMagick. Make sure they are in the path for the user running the service.

https://www.ffmpeg.org/download.html https://www.imagemagick.org/script/download.php

Update the storage settings in Iconik to say `Local keyframes` and `Local proxy` and restart the service.

Mac OS

Download and install the Iconik Storage Gateway

https://iconik.io/downloads/IconikStorageGateway-latest.pkg

It will be installed in /Library/Application Support/Cantemo/IconikStorageGateway where you will also find the config file called config.ini

Unload the service for now.

sudo /bin/launchctl unload "/Library/LaunchDaemons/com.cantemo.IconikStorageGateway.plist"

Edit config.ini to look like this (See Setting up above how to get the correct values)

[main] app-id = 22f34c6e-268e-11e7-ac53-6c4008b85488 storage-id = 15fa3f68-7693-11e7-bf0c-02420a021013 iconik-url = https://app.iconik.io/ auth-token = eyJhbGciOiJIUzI1NiIsImlhdCI6MTUwMTU3NjAyNCwiZXhwIjoxNTAxNjE5MjI0fQ.eyJpZCI6IjMzMjFkMjIyLTc2OTMtMTFlNy1hN2EyLTAyNDIwYTAyMTAxYSJ9.sgag4OQb19J9qT1IAdKZYAqbpIoai7oW60Mahye7o8s sleep-time = 10 max-transcoding-jobs = 1 log-filename = /Library/Logs/Cantemo/IconikStorageGateway/default.log

Start the service again.

sudo /bin/launchctl load "/Library/LaunchDaemons/com.cantemo.IconikStorageGateway.plist"

For following the log file, you can use the following command:

sudo tail -f /Library/Logs/Cantemo/IconikStorageGateway/default.log

Local transcoding (Optional)

For local transcoding to work you will have to install ffmpeg and ImageMagick. Make sure they are in the path for the user running the service.

https://www.ffmpeg.org/download.html https://www.imagemagick.org/script/download.php

Update the storage settings in Iconik to say `Local keyframes` and `Local proxy` and restart the service.