Storage Scanning
iconik can scan storage to see if there are any changes being made to the storage such as:
- New files
- Removed files
- Moved files
This can be done on both file storages (On your computer/server/NAS or SAN using the iconik Storage Gateway) and cloud storage types.
Any changes found that have been made to the storage can then trigger creation of new assets or updating existing assets automatically without user intervention.
Scans can be manually triggered on Cloud storage or you can setup Auto-scanning.
Scan directories can be set, and enable you to either set the entire storage for scanning or just particular directories. If you set a directory for scanning, then it will automatically recursively scan all sub-directories.
Auto Scanning Cloud Buckets
After enabling scan, you can set the scan directories that iconik will scan inside of. You can also setup Auto-Scan and the scan interval. The default is once every hour. See Storage Editing for more information on how to setup with your storage.
Please note that it should be set to give enough time to scan the storage - so a scan directory with only a few files or sub-directories can be done relatively quickly, but extremely large amounts of files and sub-directories you might want to set that higher.
If you have configured iconik to use your own cloud storage, cloud storage scanning also will incur costs as the scanning operation can count as operations on your storage by your cloud storage provider.
Auto Scanning file storage
If you Enable Scan you can setup the scan interval as well as which directories will be scanned.
The Scan Internal says how often we will initiate a Scan, and can be left on default unless the storage has many files, then you might want to increase it.
Identical files
When we find files that are identical we will register them under the same asset. It doesn't matter what storage they are on if they are found to be identical.
You can override this by setting the pattern Don't aggregate pattern, so it will always consider the files as separate assets.
It can also be configured to consider all files found as unique per storage using the Aggregate on same storage function, but if the file is found on another storage it will be aggregated into the same asset.
Other parameters
For the other parameters, such as File and Directories settings - see the Storage Editing pages for the storage that you are interested in.
Updating iconik on changes to storage
At the moment we don't consume any notifications that you can setup on storages to notify of changes to that storage (e.g.. Amazon S3 Event Notifications). However this is something that you could implement yourself, with the iconik API, the event notification from your cloud storage provider and scripts or cloud functions.
Scanning speed.
iconik scans storages by listing all files on your storage, and going into every sub-directory of your storage and compares them to what it expects to see. This operation can take some time for large file systems. For an example for half a million files it's usual to take a couple of hours to scan. If you only usually update a portion of your storage, you can use scan directories to limit the scanning to certain sub-directories of your storage and the scan time will be reduced.
Manually scanning
You can manually scan for storages that support it from the storage listing page, and the edit page.
- Click on ADMIN in the top navigation.
- Choose Storages from the left navigation bar.
- Find the storage you wish to scan.
- Click on the SCAN text.
- A scanning job will be scheduled.
The storage edit page will show the last time it was scanned.
