1. Home
  2. admin
  3. storage
  4. tutorials
  5. auto scanning

Roles

These are the roles that are needed:

  • Admin
  • Read storage
  • Write storage
  • Reindex storage

Auto-Scanning Cloud Storage.

Auto-scanning helps you work with your cloud storage by periodically read and looking for new files and changes to you cloud storage.

Advantages

  • Use Cloud storage as a place to pick up new files by iconik
  • See changes that are being made to cloud storage as they happen.
  • Only the parts of your cloud storage needing to be scanned can be configured to be scanned

Disadvantages

  • Extremely large cloud buckets can take some time to scan.
  • Cloud storage scanning also incur costs as the scanning operation can count as operations on your storage by your cloud storage provider

Planning for scanning cloud storage.

  1. Understand which of your cloud storage buckets you wish to allow this on. This should only be used on the Cloud Storage Buckets that you allow access by iconik.
  2. Work out which paths under the cloud storage you wish to have scanned.
  3. Investigate if there are any file types or directory paths that you want to have ignored from the scan (such as temporary files)
  4. Should files be checked to see if they are the same as other files?
  5. Get an understanding of how many files are in the cloud storage.
  6. Understand if you want to have any new files transcoded to create keyframes and proxies
  7. Workout who should get access to these files when they are scanned.
  8. Should collections be created for the directories that are found in the scanned storage?

Once you have gathered the required details you can then see how it's mapped to iconik's capabilities.

  1. Cloud storage scanning is configured per Cloud storage.
  2. You can define paths that the scanner is restricted to by restricting it to certain Scan Directories. These paths should be the base path - everything below it, including all directories are scanned.
  3. iconik allows for entering Regular Expression patterns of files to ignore using the Ignore File Patterns configuration. If the scanner comes across anything that matches these patterns, the file will be ignored and the scanner will move one.
  4. We offer the facility to "Consider identical files the same" by check-summing the file and see if that checksum matches any other known files. If there is a match we do not create a new asset for the file, but we map the newly discovered instance of the file to the existing asset - you will see it in the list of Files for the Asset.
  5. By default we scan once per hour - for larger storages with hundreds of thousands of directories and files or more you might wish to increase this as the scanner has to check every single one.
  6. You can set iconik up to transcode any files such as video, audio and images that the transcoders support to create keyframes and proxies for displaying in the iconik interface.
  7. Files that are found in the scanning directories that are not to be ignored will create assets in iconik - we can apply ACL templates to the newly created files to automatically set which Users and User Groups should automatically have access to these new assets.
  8. By creating collections for directories, and adding the assets created from files into these collections you can use the iconik interface to navigate through the cloud storage.

Setting up.

For each cloud storage that you wish to have scanned follow the following guide.

  1. If you wish to set the permissions on the assets that are created by newly found files you can set an ACL template. This requires having setup the ACL templates and User Groups before editing the storage. Alternatively you can setup which User Group has access by setting the Group Access setting in the Scan Settings of the storage.
  2. Visit the storage page for the cloud storage and go to Scan Settings. If you haven't yet set up the cloud storage see our guides on adding it (../adding/).
  3. Go to Scan settings and start by Enabling Scan.
  4. If you only wish to scan certain paths in your cloud storage you can enter each in the Scan Directories field. You only need to do this for the base of the path - the scanner will automatically go into any subdirectory.
  5. Set on Auto Scan.
  6. Leave the Scan Interval at one hour unless you know that you don't often write to the cloud storage, or it has a lot of files in it.
  7. If you want the scanner to ignore certain files by name enter in the pattern. This can be as simple as adding a file extension, i.e.. *.tmp.
  8. Turn on Consider identical files the same if you wish to only have one asset created for identical files. Please note that this requires reading the file to checksum it
  9. If you wish to have collections created for directories turn on Collection Directory Mapping
  10. If you want to have files transcoded, add a transcoder to the storage under the Transcode Settings Tab.

Learn more