Watchfolders

The Blackbird plc. Edge Server is a product enabling ingest and publishing of video content to the Blackbird cloud.

For single items of video content, ingesting may be performed individually. However, to enable a repeatable workflow of ingesting a supply of new content, you will find it advantageous to specify a folder into which new content is copied. Such a folder would be “watched” by the Edge Server, for the appearance of any new content. When new content is placed in the folder, this incites the Edge Server to ingest that which it has not already ingested. We describe the process to set up a watch folder, and then to use magic expressions to automate the organisation of the results.

Using the “Expert GUI” to set a watch folder

The Expert GUI is a comprehensive management interface.

Setting a watchfolder

Click the Watch Folders tab in the row of tab headers (below ‘Default account’ and ‘Default folder’ menus). Double-clicking on a row entry under the Files column will comprehensively list the status of your files.

Pressing the Browse button will show a file picker dialog, from which you may select file(s) from your Mac, or from connected volumes. To ingest the contents of a folder, go to the folder and press Open. Otherwise, for individual files, see ‘Drag and Drop’ (described later).

  • Double click the entry in the Account column, an already specified account name will be presented in a menu (along with others which may apply).

  • Click the blank field in the Folder column and type the relevant folder name.

  • The Imm column contains a box which you should tick, which sets ingest to ‘immediate’. The thumbnail and name will immediately appear in Blackbird itself.

  • Select the appropriate resolution and framerate (if known) from the Ingest column.

  • Click the empty entry in the Kind column, in the resulting menu select ‘plain’.

  • Click to tick the box in the On column near the left, then over on the far right, click the Check now button.

Progress view in the Expert GUI

The Expert GUI may be invoked to see greater technical detail of job progress.

The Local Disk tab shows status of the Edge server’s own work space as it ingests. Files that are part of ingest jobs are indicated under the File Queue tab. Files with unexpected frame rates are indicated here.

  • Clicking the Show Files button will display a list of files with their filename in the leftmost column.

  • The columns Q state and Message display information regarding ingesting and ascertained frame rate.

  • Files indicated as ‘to ingest’ under Q state have been accepted as the correct frame rate.

    • Files that do not display this and instead display their true frame rate under Message require your attention.

To monitor progress, go to the Local Disk tab to view the upload percentage on the leftmost column, once it has updated.

Using “Magic Expressions” to organise your tasks and save time

At the Expert GUI you will notice that many of the columns of data are related to metadata (that is, data about the media assets). One of the more time consuming tasks in a production workflow is repetitively putting things into appropriate folders. As it would be valuable to be able to sort the product of an ingest task into appropriate folders automatically, there is a special feature made available to advanced users like yourself.

With “magic expressions” you can get large amounts of ingested material sorted out into a useful folder structure completely automatically. They let you do things like take the separate parts of your disklabels and lay the material out in different folder levels – for example by team on an outer level and then inside that by date.

Notice that this sorting is something being done by a computer following strict rules, so the more consistent your metadata is the better. The expressions can do things like split out the 3rd and 4th, or perhaps last 2 characters of something, or pick out the second section separated by eg underscore "_" characters. They have access to any metadata that is in the file that the Edge server is ingesting, and it’s filename and existing directory layout. You can also have alternate layouts in the case of different metadata, eg putting anything missing a disk label in a “no disk label” folder!

When you are getting things set up, it is usually a good idea to ingest a few day’s worth of your material first, and only then try to set up or have set up the absolutely perfect organize expression. The Edge server has the ability to move already ingested material about to suit a changed “magic expression” (the Re-organize button). But if you plan to do this, don’t manually move material about before that as that will get un-done! (Copying it is ok of course).

When ingesting material directly from an avid workspace, there is a big trap to watch for – Blackbird Edge only sees the original disk label, and not any changes that you do in Avid after the original ingest, since these are only in Avid’s database and do not change the mxf file. Avid does make available the original source file path that it ingested from in the metadata sourcelocator.

Magic expressions are “magic” in the sense that you don’t need to understand how they work, you can just copy the necessary magic words to do particular things, and our support team is always available to help. Of course, it is no secret how they work, so if you or someone on your team is a bit of a programmer, feel free to experiment, there is the “Re-organize” feature available to let you change the expression and get material moved around to see the result. It is a programming language, so sometimes it will just move all your material into a directory named for an error message! But it can move it out again just as easily.

The true “magic expressions” are the ones starting with =. There is a simpler version just using the byxxx words, which let you do a few straightforward things like directly copy the whole layout of the ingested files using eg "MyIngests/bypath".

Here are a few simple examples to try out, which may be of value to you. In these examples, we have a watchfolder which, in the Expert GUI, we have set a name of “MyIngests” (this of course could be anything you wish to call it).

Magic Expressions in the Expert GUI Folder column

The “magic expression” itself is typed directly into the field under the Folder column of the Expert GUI – simply click on the name of the job you are working on, in that column, and it will turn into an editable text field.

MyIngests/bypath

This will replicate the folder structure on your watchfolder (which may even be an external disk). Your ingested result will be placed and nested in named folders which match the arrangement of the source disk.

MyIngests/bydisk

This will set a destination folder with the same name as the original Avid Import Disk Label (or if this is not set, the reelname). This of course is mainly used when the source came from an Avid environment.

="MyIngests/"+disk[0:4]

This will take (in this example) the first four characters of the disk label, to use as the destination folder name.

The first two examples use the “shortcut” form, of bypath and bydisk, and consequently do not require an equals sign = at the start. The third example uses the full “magic expression” form, which does require the equals. You should use one approach or the other, but not both in combination.

Magic Expressions in the Expert GUI Metadata column

Unlike the above examples, which result in a specifically named folder, the following “magic expressions” work as filters. In other words, if you specify a condition or requirement among a group of files, and this condition turns out to be true (some or all of the files satisfy that condition) then the ingest task will only apply to those files included within the selection. Files which were not selected by the specified condition are simply not acted upon. In general, these selections are based upon the metadata of a file, for example the date or time of creation. Here is an example of a “magic expression” which can select and ingest only files in the specified watch folder which have a file creation time of or after a specific time.

=filetime>time(12,30,00)

This will select files which have a creation time on, or after, half past midday. The hours slot is considered to use the 24-hour clock, hence, 12 here is noon (plus, 30 minutes and zero seconds). The “greater than” > character is treated here as equals or greater than the given time.

=filedate>date(2018,6,18)

This will select files with a creation date after (but not on) the eighteenth of June, in the year 2018. The “greater than” > character is treated literally as greater than the given date. Hence, if you want to select files that include the date mentioned above, specify the day before that.

All files not selected by these metadata filter “magic expressions” will simply be left alone and not ingested. It may transpire that you make a filter which selects no files at all to be ingested (or alternatively, all of them, without performing any useful filtering). In such cases you would edit the filter expression, or the “magic expression” and start again. However, any time you wish to change the values of these filter expressions, always click on the Forget button before re-checking the watch folder.

Avoiding unwanted duplicate ingest jobs

Watch folders are file-oriented, additionally considering the entire file path of directories through which we reach the file as the unique distinguishing aspect which identifies a particular file. Initially it will make note of the existence of a file, and only once. If the file is subsequently moved to a different folder, the file will be re-scanned for metadata, and then grouped into a single job item (along with any other new files encountered together) to add to the ingest queue.

The ingest queue, on the other hand, is oriented to jobs (ingest tasks). A queued item may be one file or may be a group of files encountered together during the watch folder scanning. The ingest queue is aware of duplicated jobs, indicating a special state dup when a queue item (an ingest job) has already been ingested to that account (either by this Edge Server, or any other).

If you require it, you are able to force ingest again despite the dup – either by setting the over-ride constantly as a watch folder setting, or by manual over-ride from the queue.

There is a same item check, which uses a unique identifier called materialuid, if one is available on that media. If this is not available, the system uses disklabel + reelname + timecode instead, to arrive at a usable form of unique identifier.

This means that if an Avid system is used to transcode the same asset twice, and the camera originally gave no materialuid metadata, this counts as two different assets – which will be ingested twice.

Consider the scanning delay

It is also important to bear in mind that all accounts feature an intentional delay in updating any scanning for new material, of between five to ten minutes. If multiple watchfolder setups from more than one Edge Server result in the same file or folder being scanned at more or less the same time (that is, within the same five to ten minute window) it may result in the same file being ingested more than once. One Edge Server will queue it for ingest, and another Edge Server also reaching the same asset will not realise that it would potentially be considered a dup due to the delay.

Another aspect to consider is the moving of files within a watch folder, where the files have already been added to the ingest queue, yet have not begun actually being ingested. If such assets are moved elsewhere during this brief duration, they will be flagged as missing. For example, perhaps they were on a watch folder which is on an external disk, and that disk was unplugged at an inopportune time. If the assets are returned to the place they were in when scanned, they will be picked up in due course and ingested as normal.

You can transfer a watch folder to a different Edge Server.

  1. Disable it on the original Edge Server

  2. Wait a while (for the scanning delay to be safely behind you)

  3. Also, wait for the ingest queue to complete, if it is in progress

  4. Enable the watch folder again on the substitute Edge Server

  5. Clean up all the dup entries from the queue, after the first scan

Appendix

Further “magic expression” values

These “magic expression” values for the file dates are also available:

filedatetime

A “Python” datetime.datetime object in the edge servers’s local timezone for the latest date of any of the item’s files

filedate

Date for filedatetime

filetime

Time for filedatetime

datetime

“Python” datetime.datetime class

date

“Python” datetime.date class

time

“Python” datetime.time class

localdatetime

A function to construct a datetime in the edge server’s local timezone (accepts y,m,d[h,m,s,[us]]).

tzlocal

A function to return the edge server’s local timezone (tzdata object)

str

“Python” str function, used to convert filedate* to a string

Also, “magic expression” examples using these:

folder:

="Myingests/"+str(filedate)

metadata_filter:

=filedate>=date(2018,2,3)

References to “Python” indicate use of the “Python programming language” syntax, as detailed in most “Python programming language” reference guides.