iconik Hybrid Cloud Architecture

iconik is designed to be a hybrid cloud solution which manages assets both in the cloud and on local on-premise storages. This design allows you to bring both your own cloud bucket as well as have iconik manage the storage infrastructure you already have invested in.

iconik Storage Gateway

The iconik Storage Gateway (ISG) is a software service which monitors an on-premise storage and indexes the contents of the storage in iconik. The ISG is supported on Windows, MacOS, CentOS and Ubuntu.

The iconik Storage Gateway can be deployed in several different ways. It can be deployed as part of a larger installation where multiple storage gateways monitor different storage areas in a large shared storage or it can be deployed on a single computer and only monitor the local disk. Both of these scenarios are outlined in the diagram above.

Storage Scanner

The ISG periodically scans the monitored storage. When it discovers a new file it starts by calculating a checksum of the file to discover if this is a file which already is known to the system. If it finds that an asset already exists with the same checksum then the newly discovered files will be associated to the existing asset. If the ISG cannot find an existing asset then a new asset is created in iconik and if the file is a media file then a low resolution proxy is created and uploaded to iconik to allow users to view a representation of the asset in the iconik web interface.

The ISG continues to scan the storage and if a file it has previously detected cannot be found then the file will be marked as missing and any file operations iconik or the gateway wants to perform will be directed to other copies of the same file on other storages.

Transcoder

The iconik default transcoder is built on the Open Source software solutions FFMPEG and ImageMagick. These allow the ISG to generate proxy formats suitable for viewing in a web application such as iconik. These proxies are also used in the proxy workflow in our Adobe Premiere integration.

Transcoder integrations

The ISG can also be used to integrate with Telestream Vantage. This allow you to use existing transcoding farms you may already have invested in an can also be used to scale the transcoding out to multiple servers. In this scenario, the ISG and the transcoders both have to have access to the same underlying storage system as the ISG will control the transcoders to generate proxy files. Once the transcode job completes the ISG will pick up the file and upload it to the cloud via the public iconik APIs.

Communication

All communication from the iconik Storage Gateway to iconik is performed over https with TLS 1.2, and all network traffic to iconik is initiated from the iconik Storage Gateway. The main benefit of this design is that no incoming ports have to be opened in the local firewall. The only required port is outgoing https on port 443, and the ability to lookup DNS entries.

File transfers

The ISG can transfer files to and from a cloud storage. This is done using pre-signed URLs which the iconik cloud service creates using the access credentials configured in the storage settings. By using pre-signed URLs, the ISG only gets access to perform the specific operation on the specific file, without the ability to access any other files on the storage. Files are uploaded directly from the ISG to to cloud storage without passing through the iconik cloud infrastructure.

Transfer flow

The following diagram shows the flow of a cloud bucket upload.

1. User or automation initiates a transfer.
2. On-premise ISG periodically makes a request to the iconik APIs to fetch the queued transfer requests. These requests are queued internally in the ISG's database and are processed in order.
3. When the transfer is set to start, the ISG makes a call to iconik to initiate the transfer, creates a job and obtains a time-limited pre-signed URL for the destination storage.
4. The ISG uploads the file directly to the destination storage. Exactly how this is done depends on the type of destination storage, but in most cases this is a concurrent multi-part upload using HTTPS.
5. Once the file has been successfully transferred the ISG issues another set of calls to the iconik APIs to finalize the transfer, close the file record and mark the transfer job as completed.

The above process is also used for transfers from a cloud bucket to an ISG, but the actual file transfer in step 4 happens in the opposite direction.

ISG to ISG transfers

The default behaviour when transferring files from one ISG storage to another ISG storage is to first upload the file to the sending ISGs default cloud bucket and then downloading it to the receiving ISG. Depending on the network topology you can configure you iconik Storage Gateways to transfer files directly from ISG to ISG instead of transferring it via the cloud.

The ISG comes with an embedded HTTPS webserver which is used for the transfer.

This transfer is performed via HTTPS with a self-signed certificate generated by the ISG.

Transfer flow

The following diagram shows the flow of a direct transfer

1. User or automation initiates a transfer.
2. Client ISG makes a periodic request to iconik to request all pending transfers. Each transfer is identified using a unique transfer id.
3. The Client ISG determines that the transfer is a candidate for a direct transfer and makes a request to iconik for a transfer URL. This URL is signed with a symmetric encryption key which is stored in the iconik database and is never sent to the client ISG. iconik also responds with the https certificate of the Host ISG.
4. Client makes a request to the Host ISG, requesting the file. This is done using a http range request and multiple requests are performed in parallel to improve transfer performance. The Host's HTTPS certificate is verified against the certificate received from iconik in step 3.
5. The Host ISG passes the request parameters to iconik which verifies that the signature is valid using the key and the transfer id.
6. If the validation is successful then the Host starts to respond to the HTTPS requests from step 4 until the transfer is completed.
7. Once all the range requests have completed a call is made to iconik to finalize the transfer and make sure the file records for the asset is updated to reflect that the file now resides on the client storage.

Security

Because the ISG opens up a webserver on the customer's internal network we have to make sure that only legitimate transfers are allowed to transfer files. The two ISGs which are involved in the transfer need to authenticate each other. Two methods are used to accomplish this.

The Client ISG authenticates the Host ISG via its https certificate. This certificate is generated by the Host ISG when it starts and the certificate is registered in iconik as belonging to the Host ISG. This certificate is passed to the Client ISG as part of the transfer initialization (3) so when the Client ISG connects to the Host ISG via HTTPS (4) then the Client ISG knows which certificate to expect and can fail the transfer if the certificate the Host presents doesn't match the expected one.

The Host authenticates the Client via a signature which is included in the query parameters of the request URL. When iconik initiates a transfer (1), a random symmetric key is generated and stored in iconik's database. This key is then used to sign the transfer URL and parameters which are passed to the Client ISG (3). The Client ISG then passes this URL to the Host ISG as part of the request to fetch the file (4). The Host ISG cannot verify the validity of the signature itself so it makes a call to iconik to verify the signature (5) and iconik uses the key it already has stored to verify that the signature is correct. If iconik responds with a positive response then the Host ISG returns the file to the Client ISG.

iconik Adobe Panel

An important part of the iconik hybrid cloud solution is the integration with the Adobe Creative Cloud suite. The iconik Panel can be downloaded from Adobe Exchange and allows the user to access the iconik interface from within the Creative applications. The panel runs essentially the same code as the iconik web interface, with minor modifications to work better within the restrictions of the creative applications. As with the iconik web interface, all communication with iconik happens via the public APIs, and communication with the Adobe Creative applications is done through the Common Extensibility Platform

Integration with iconik Storage Gateway

The Adobe Panel can be used stand-alone, in which case media files the user accesses will be transferred to and from the cloud buckets. It can also be used together with a locally installed iconik Storage Gateway. In this operational mode, the ISG is responsible for scanning and indexing the storage as well as uploading media files to the cloud. When the user imports an asset into the project the file is accessed directly from the shared storage without any need to download the file.