iconik Hybrid Cloud Architecture

iconik is designed to be a hybrid cloud solution which manages assets both in the cloud and on local on-premise storages. This design allows you to bring both your own cloud bucket as well as have iconik manage the storage infrastructure you already have invested in.

iconik Storage Gateway

The iconik Storage Gateway (ISG) is a software service which monitors an on-premise storage and indexes the contents of the storage in iconik. The ISG is supported on Windows, MacOS, CentOS and Ubuntu.

The iconik Storage Gateway can be deployed in several different ways. It can be deployed as part of a larger installation where multiple storage gateways monitor different storage areas in a large shared storage or it can be deployed on a single computer and only monitor the local disk. Both of these scenarios are outlined in the diagram above.

Storage Scanner

The ISG periodically scans the monitored storage. When it discovers a new file it starts by calculating a checksum of the file to discover if this is a file which already is known to the system. If it finds that an asset already exists with the same checksum then the newly discovered files will be associated to the existing asset. If the ISG cannot find an existing asset then a new asset is created in iconik and if the file is a media file then a low resolution proxy is created and uploaded to iconik to allow users to view a representation of the asset in the iconik web interface.

The ISG continues to scan the storage and if a file it has previously detected cannot be found then the file will be marked as missing and any file operations iconik or the gateway wants to perform will be directed to other copies of the same file on other storages.

Transcoder

The iconik default transcoder is built on the Open Source software solutions FFMPEG and ImageMagick. These allow the ISG to generate proxy formats suitable for viewing in a web application such as iconik. These proxies are also used in the proxy workflow in our Adobe Premiere integration.

Transcoder integrations

The ISG can also be used to integrate with Telestream Vantage. This allow you to use existing transcoding farms you may already have invested in an can also be used to scale the transcoding out to multiple servers. In this scenario, the ISG and the transcoders both have to have access to the same underlying storage system as the ISG will control the transcoders to generate proxy files. Once the transcode job completes the ISG will pick up the file and upload it to the cloud via the public iconik APIs.

Communication

All communication from the iconik Storage Gateway to iconik is performed over https with TLS 1.2, and all network traffic to iconik is initiated from the iconik Storage Gateway. The main benefit of this design is that no incoming ports have to be opened in the local firewall. The only required port is outgoing https on port 443, and the ability to lookup DNS entries.

ISG to ISG transfers

The default behaviour when transferring files from one ISG storage to another ISG storage is to first upload the file to the sending ISGs default cloud bucket and then downloading it to the receiving ISG. Depending on the network topology you can configure you iconik Storage Gateways to transfer files directly from ISG to ISG instead of transferring it via the cloud.

The ISG comes with an embedded HTTPS webserver which is used for the transfer.

This transfer is performed via HTTPS with a self-signed certificate generated by the ISG.

Transfer flow

The following diagram shows the flow of a direct transfer

1. User or automation initiates a transfer.
2. Client ISG makes a periodic request to iconik to request all pending transfers. Each transfer is identified using a unique transfer id.
3. The Client ISG determines that the transfer is a candidate for a direct transfer and makes a request to iconik for a transfer URL. This URL is signed with a symmetric encryption key which is stored in the iconik database and is never sent to the client ISG. iconik also responds with the https certificate of the Host ISG.
4. Client makes a request to the Host ISG, requesting the file. This is done using a http range request and multiple requests are performed in parallel to improve transfer performance. The Host's HTTPS certificate is verified against the certificate received from iconik in step 3.
5. The Host ISG passes the request parameters to iconik which verifies that the signature is valid using the key and the transfer id.
6. If the validation is successful then the Host starts to respond to the HTTPS requests from step 4 until the transfer is completed.
7. Once all the range requests have completed a call is made to iconik to finalize the transfer and make sure the file records for the asset is updated to reflect that the file now resides on the client storage.

Security

Because the ISG opens up a webserver on the customer's internal network we have to make sure that only legitimate transfers are allowed to transfer files. The two ISGs which are involved in the transfer need to authenticate each other. Two methods are used to accomplish this.

The Client ISG authenticates the Host ISG via its https certificate. This certificate is generated by the Host ISG when it starts and the certificate is registered in iconik as belonging to the Host ISG. This certificate is passed to the Client ISG as part of the transfer initialization (3) so when the Client ISG connects to the Host ISG via HTTPS (4) then the Client ISG knows which certificate to expect and can fail the transfer if the certificate the Host presents doesn't match the expected one.

The Host authenticates the Client via a signature which is included in the query parameters of the request URL. When iconik initiates a transfer (1), a random symmetric key is generated and stored in iconik's database. This key is then used to sign the transfer URL and parameters which are passed to the Client ISG (3). The Client ISG then passes this URL to the Host ISG as part of the request to fetch the file (4). The Host ISG cannot verify the validity of the signature itself so it makes a call to iconik to verify the signature (5) and iconik uses the key it already has stored to verify that the signature is correct. If iconik responds with a positive response then the Host ISG returns the file to the Client ISG.

iconik Adobe Panel

An important part of the iconik hybrid cloud solution is the integration with the Adobe Creative Cloud suite. The iconik Panel can be downloaded from Adobe Exchange and allows the user to access the iconik interface from within the Creative applications. The panel runs essentially the same code as the iconik web interface, with minor modifications to work better within the restrictions of the creative applications. As with the iconik web interface, all communication with iconik happens via the public APIs, and communication with the Adobe Creative applications is done through the Common Extensibility Platform

Integration with iconik Storage Gateway

The Adobe Panel can be used stand-alone, in which case media files the user accesses will be transferred to and from the cloud buckets. It can also be used together with a locally installed iconik Storage Gateway. In this operational mode, the ISG is responsible for scanning and indexing the storage as well as uploading media files to the cloud. When the user imports an asset into the project the file is accessed directly from the shared storage without any need to download the file.