ISG Advanced Options (And Edge Transcoder)

The iconik Storage Gateway has a number of advanced options that you can tune by editing the configuration file config.ini. Most of these options also apply to the iconik Edge Transcoder, except the ones related to scanning

• Back up config.ini before you make these changes.
• Configuration options are key value pairs such as proxy-resolution=1920x1080
• Configuration options should be on individual lines.
• Restart the ISG after making the changes

config.ini location

Please make sure if you are editing on Windows the line endings are of a Unix type

Setting Proxy Format Specification

It's possible to change the proxy format that the ISG will generate for video media. Please note that setting the bitrate too high or low can have a negative impact on the perceived performance of the Web Interface.

Proxy resolution:

By setting the key proxy-resolution you can define the max-dimensions of the proxy created.

• Minimum value: 640x360
• Default value: 1280x720
• Maximum value: 1920x1080

For example:

proxy-resolution=1920x1080

Proxy bitrate

By setting the key proxy-bitrate you can define the max bitrate of the proxy created. Please be aware that setting this higher can cause large file sizes for your proxy.

• Minimum value: 200000
• Default value: 3000000
• Maximum value: 10000000

For example:

proxy-bitrate=4000000

Controlling file transfers and transcodes

Number of parallel uploads and downloads

By setting the key file-upload-parallel-uploads-num you can control how many simultaneous uploads the ISG will do. The default is 4 By setting the key file-upload-parallel-uploads-threads-num you can control how many parallel chunks for each upload is used. The default is 5 By setting the key upload-chunk-size you can control the size of parallel chunks for each upload in bytes. The default is 104857600 (100 Mb)

file-upload-parallel-uploads-num = 4

file-upload-parallel-uploads-threads-num = 5

upload-chunk-size = 104857600

By setting the key file-download-parallel-downloads-num you can control how many simultaneous downloads the ISG will do. The default is 4 By setting the key file-download-parallel-downloads-threads-num you can control how many parallel chunks for each download is used. The default is 5 By setting the key download-chunk-size you can control the size of parallel chunks for each download in bytes. The default is 104857600 (100 Mb)

file-download-parallel-downloads-num = 4

file-download-parallel-downloads-threads-num = 5

download-chunk-size = 104857600

Number of parallel transcodes

By setting the key max-transcoding-jobs you can control the number of simultaneous local transcodes.

max-transcoding-jobs = 2

This should be roughly the number of processing cores of your machine divided by 4. So if you 8 cores in your processor, the number should be 2.

Enabling debug

Debug is enabled by setting the option debug to true

debug = true

Mediainfo timeouts

If you get timeouts reading media info you can increase that, the default is 60s

extract-mediainfo-timeout = 180

Control the number of checksum workers

Checksumming is both CPU and I/O intense. You can control the number of parallel workers doing checksumming. The default is the number of CPUs on your server

checksum-max-workers = 4

You can also enable and disable it (but it is required for "Consider identical files the same")

enable-checksum-calculation = true

Enable checksum calculation based on modification time

The ISG will consider a file changed if its size changed, updating its checksum if checksum calculation is enabled. If your content changes without changing file size you can trigger checksum calculation by enabling modification time comparison.

checksum-on-modification-time-change = true

Disable the UUID-suffix on uploaded files

By default the ISG will add a UUID to the end of the file name when uploading to a cloud storage to make sure it does not conflict with any existing files. This can be disabled if you know that only this ISG will write to that cloud storage.

auto-generate-filename-for-remote-upload = false

Overriding format detection

Some times you might have files that gets identified as the wrong type. You can override this by file extension by providing a list:

image-extensions-override=.cr3,.raf,.psd,.tiff

video-extensions-override=.mp4,.mxf

audio-extensions-override=.wav,.m4a

This means that regardless of what mediainfo returns they will be sent to the transcoder that handles the type of format you have specified.

Some formats require to flatten layers, this list can be modified.

imagemagick-layer-extensions=.ai,.tif,.tiff,.png,.psd,.pdf,.psb,.gif,.eps

For Edge transcoder some formats also require downloads, since they can't be read over https.

image-extensions-requiring-download=.rw2,.arw,.dng,.pdf,.raf,.cr3,.cr2,.heic

Overriding proxy and keyframe storage

You can override the proxy and keyframe storage used for an ISG by putting the follwing in the config.ini file:

proxy-storage-id=<storage_id>

keyframe-storage-id=<storage_id>

ImageMagick Custom Path

If you have installed ImageMagick outside of the system path it can be useful to specify a path to imageMagick configuration files:

image-magick-config = /custom/path

This path should be where policy.xml, delegates.xml etc. is located

Use embedded JPGs for RAW formats

Many RAW formats contain an embedded image that ISG will extract and use to create a proxy. Extracting the embedded image requires exiftool to be installed and is enabled by default. You can disable this behaviour by setting

use-embedded-raw-images = false

You can configure which file extensions the ISG considers RAW formats by this setting (default in the example)

raw-image-extensions = .rw2,.arw,.dng,.cr2,.cr3,.raf,.nef,.3fr,.x3f,.nrw

Apply custom RGB conversion profiles (ICC)

If you have for example TIFF files in CMYK that requires a specific ICC to convert to RGB, you can specify that. You can specify any number of icc files in order separated by semicolon. Note that the value is a comma separated list of extension=path1;path2

apply-image-convert-profile=.tif=/profiles/original1.icc;/profiles/export1.icc,.tiff=/profiles/original1.icc;/profiles/export1.icc

WAL mode

WAL can improve performance on large ISG installation. It is controlled by this setting.

sqlite-journal-mode = wal

To turn it off set

sqlite-journal-mode = delete

Video encoder

You can change the default h.264 encoder. For example on Apple silicon you can try this for better performance. Note that this also requires you to set the resolution and bitrate on the attached transcoder to get decent quality

video-encoder=h264_videotoolbox

Video encoder for edit proxies

You can set custom video encoder for edit proxies

edit-proxy-video-encoder=libx264

Audio encoder for edit proxies

Default is AAC (aac) but for some formats it might be better to use PCM (pcm_s24le)

edit-proxy-audio-encoder=pcm_s24le

Caching proxy for Edge Transcoder

Edge Transcoder can make use of a local caching proxy to improve performance.

use-file-cache-proxy = true

Start caching proxy instructions for Centos7 and Ubuntu.

White balance adjustment for Raw formats

You can try to add this if you have ImageMagick7 with libraw. Note that it ONLY works on IM7 compiled with libraw. This options tries to use the camera settings for White Balance

imagemagick-white-balance-extensions=.cr3,.nef

Additional Options when using LucidLink Filespaces on the ISG host

The LucidLink remote upload indicator API can be used to detect when a file is ready for ingest. In order to use it with ISG your LucidLink instance must meet the following requirements:

• LucidLink Client version 2.1.4246 or above required.
• LucidLink 2.0 Filespace or above required

To enable remote upload indicator API support in ISG set

enable-lucidlink-upload-api=true

When using multiple Filespaces the ISG needs to know the instance ids in use, provided as a comma-separated list. LucidApp uses id 0 by default.

lucidlink-instances=0,1,2

The ISG will try to detect what mount points are in use for each LucidLink instance. You can also hard-code the filespace mount points the ISG uses with the remote upload indicator API. The order of the list of mount points must match the lucidlink-instances list if using multiple filespaces.

lucidlink-mount-points=/Volumes/fs1,/Volumes/fs2,/Volumes/fs3

Custom edit proxy suffix

You can specify a suffix that will be on the filename of already generated proxy files which are stored next to the original file that will be pattern matched and stored as an Edit Proxy by iconik.

custom-proxy-suffixes=_customeditproxy

You can change the suffix that is added to the edit proxy created by local transcoders.

• Default value: _editproxy

edit-proxy-suffix=_editproxy

Troubleshooting performance

You can control max amount of parallel ingests for new files.

ingest-concurrency-value = 2

You can control max amount of parallel directory scans.

scanner-concurrency-value = 2

Increasing these values might provide a higher ingest and scan speed, but the speed can be limited by the available host resources, including network and storage performance.

You can enable verbose performance logging.

debug-performance = true

Learn more

• iconik Storage Gateway Overview