Tutorial for the DirectorySpyApp

Motivation

The DirectorySpyApp is designed to keep track of the files in a directory and notify the user if a new file (with a specific filename pattern) has been created or changed. If configured to do so, it will also supply the image data of the latest image as well as the filename.

Note

While the use of different Parameters is handled in different sections of this manual, they must all be combined to use all capabilities of the DirectorySpyApp.

Setup of the DirectorySpyApp

This section describes the different input Parameters that can be used and gives an overview of the DirectorySpyApp Parameters at the end of the section.

Selection of input data

First of all, the user must specify the directory_path Parameter to define the working directory. In addition, the user must decide whether the app will scan for all new files the selected directory or only for files matching a specific filename pattern. In the latter case, the filename_pattern Parameter is used and must include the filename pattern with hashes “#” for the counting variable.

The file finding behaviour is controlled through the scan_for_all Parameter. If set to True, the app will look for all files in a directory. If False, it will only look for incremental counts according to the filename pattern and the current index.

Note

Using the scan_for_all=True setting will be slow in directories with many files and it is not recommended.

The filename_pattern Parameter only includes the filename with hash characters as wildcards. The number of wildcard characters must correspond to the length of the numbers to be replaced and it will be filled with leading zeros.

Note

Only a single set of wildcars is accepted. Please leave all other numbers in place.

If HDF5 files are used, the dataset to use must be specified with the hdf5_key. The frame cannot be selected as the DirectorySpyApp will always show the latest frame.

See Complete list of CompositeCreatorApp Parameters for the detailed list of all Parameters.

Example 1: Scanning for all files

For the first example, we want to scan for all new files in the /scratch/scan_42/ directory. We expect Hdf5 files with the data in the entry/other_data/data/ dataset.

>>> import pydidas
>>> app = pydidas.apps.DirectorySpyApp()
>>> app.set_param_value('scan_for-all', True)
>>> app.set_param_value('directory_path', '/scratch/scan_42/')
>>> app.set_param_value('hdf5_key', '/entry/other_data/data')

Example 2: Scanning for new tiff files

For this example, we want to load Tiff files from the /scratch/test_scan/ directory. The files are named test_scan_01_0001.tiff, test_scan_01_0002.tiff etc.

>>> import pydidas
>>> app = pydidas.apps.DirectorySpyApp()
# scan_for_all is False by default, no need to set it.
>>> app.set_param_value('directory_path', '/scratch/test_scan')
>>> app.set_param_value('filename_pattern', 'test_scan_01_####.tiff')

Detector mask and background image

Using a mask file

A mask file can be used by activating the use_det_mask Parameter. This will instruct the app to apply the detector mask to the data frame.

To modify the detector mask used by the DirectorySpyApp, set the detector_mask_file Parameter to point to the mask file. The value taken for masked pixels is controlled by the det_mask_val Parameter. The default value is 0.

>>> import pydidas
>>> app = pydidas.apps.DirectorySpyApp()
>>> app.set_param_value("use_detector_mask", True)
>>> app.set_param_value("detector_mask_file", '/home/user/data/detector_mask.npy')

Using a background file

Usage of a background file (which will be subtracted from all frames) can be activated by setting the use_bg_file Parameter to True.

The background file itself can be selected by specifying the bg_file Parameter. If a hdf5 file is selected, the dataset and frame can be given by the bg_hdf5_key and bg_hdf5_frame Parameters. These values default to entry/data/data and 0, respectively.

As example, let us use the first frame (i.e. zero) from the /scratch/scan_42/test.h5df5 file and the entry/detector/data dataset:

>>> import pydidas
>>> app = pydidas.apps.DirectorySpyApp()
>>> app.set_param_value('use_bg_file', True)
>>> app.set_param_value('bg_file', '/scratch/scan_42/test.h5df5')
>>> app.set_param_value('bg_hdf5_key', 'entry/detector/data')
# Because the bg_hdf5_frame defaults to 0, this Parameter does not need to
# be modified:
>>> app.get_param_value('bg_hdf5_frame')
0

Running the DirectorySpyApp

Once configured, the DirectorySpyApp is run like any pydidas app, as described in detail in Running pydidas applications.

Warning

Because the DirectorySpyApp does not use tasks and is running indefinitely until stopped, it cannot be run serially.

To run it utilizing parallelization, set up an AppRunner and use the start method:

>>> app = pydidas.apps.DirectorySpyApp()
>>> runner = pydidas.multiprocessing.AppRunner(app)
>>> runner.start()

Accessing results

DirectorySpyApp results can be only be accessed indirectly within Python.

Accessing results within Python

The results can be accessed through the image, filename and image_metadata properties. Note that this requires to connect the AppRunner sig_results with the local app’s multiprocessing_store_results method:

>>> app = pydidas.apps.DirectorySpyApp()
>>> runner = pydidas.multiprocessing.AppRunner(app)
>>> runner.sig_results.connect(app.multiprocessing_store_results)
>>> runner.start()
>>> app.image
array([[0.98215663, 0.30682687, 0.21160315, ..., 0.2604671 , 0.59461537,
    0.09863754],
   [0.51141869, 0.32276036, 0.43406916, ..., 0.02741798, 0.91533116,
    0.79145334],
   [0.1881628 , 0.4708237 , 0.14207525, ..., 0.26664729, 0.68337244,
    0.83566994],
   ...,
   [0.6985084 , 0.58230171, 0.11641333, ..., 0.3299515 , 0.29834082,
    0.19949315],
   [0.54581434, 0.96941275, 0.21216339, ..., 0.26659825, 0.13700608,
    0.01721194],
   [0.74946649, 0.24262777, 0.94001868, ..., 0.29572706, 0.68824381,
    0.61529555]])
>>> app.filename
/scratch/test_scan/test_scan_01_0004.tiff

DirectorySpyApp Parameters

scan_for_all (type: bool, default: False)

Flag to toggle scanning for all new files or only for files matching the input pattern (defined with the Parameter filename_pattern).

filename_pattern (type: pathlib.Path, default: <empty Path>)

The pattern of the filename. Use hashes “#” for wildcards which will be filled in with numbers. This Parameter must be set if scan_for_all is False.

directory_path (type: pathlib.Path, default: <empty Path>)

The absolute path of the directory to be used.

hdf5_key (type: Hdf5key, default: entry/data/data)

Used only for hdf5 files: The dataset key.

use_global_det_mask (type: bool, default: True)

Keyword to enable or disable using the global detector mask as defined by the global mask file and mask value.

use_bg_file (type: bool, default: False)

Keyword to toggle usage of background subtraction.

bg_file (type: pathlib.Path, default: <empty Path>)

The name of the file used for background correction.

bg_hdf5_key (type: Hdf5key, default: entry/data/data)

Required for hdf5 background image files: The dataset key with the image for the background file.

bg_hdf5_frame (type: int, default: 0)

Required for hdf5 background image files: The image number of the background image in the dataset.