pydidas general command line tutorial#

Accessing Parameters#

All pydidas objects handle user variables in the form of Parameters which enforces type checks where required. These objects are organized in ParameterCollections which allow direct access to the Parameter values. Usually, users do not need to create or manage Parameters but only modify their values.

Useful methods of ParameterCollection#

For the full ParameterCollections method documentation, please refer to the class documentation. However, users will probably manage with a handful of methods:

get_param_keys()

Get all the reference keys to access the respective parameters.

get_param_value(key)

Get the value of the Parameter referenced by key.

get_param_values_as_dict()

Get the value of all stored Parameters in form of a dictionary with the Parameter reference keys as dict keys and the Parameter values as dict values.

set_param_value(key, value)

Set the value of the Parameter referenced by key to the given value.

Tip

The described methods are universal and apply to all pydidas objects with Parameters.

Examples#

All examples will use the DiffractionExperimentContext object (please see below for a description of the object) and the examples will only cover the code bases, not the use case.

First, let us create the object called exp

>>> import pydidas
>>> exp = pydidas.contexts.DiffractionExperimentContext()

The object exp will be used in all examples below.

  1. Get all Parameter keys contained in exp:

    >>> exp.get_param_keys()
['xray_wavelength',
 'xray_energy',
 'detector_name',
 'detector_npixx',
 'detector_npixy',
 'detector_pxsizex',
 'detector_pxsizey',
 'detector_mask_file',
 'detector_dist',
 'detector_poni1',
 'detector_poni2',
 'detector_rot1',
 'detector_rot2',
 'detector_rot3']

2. Get the value of the Parameter xray_energy

>>> exp.get_param_value('xray_energy')
15.0

3. Get the values of all Parameters as a dictionary to process further:

>>> params = exp.get_param_values_as_dict()
>>> params
{'xray_wavelength': 0.8265613228880018,
 'xray_energy': 15.0,
 'detector_name': 'Eiger 9M',
 'detector_npixx': 3110,
 'detector_npixy': 3269,
 'detector_sizex': 7.5e-05,
 'detector_sizey': 7.5e-05,
 'detector_mask_file': Path('.'),
 'detector_dist': 0.23561364873702045,
 'detector_poni1': 0.11575233539615679,
 'detector_poni2': 0.12393982882406686,
 'detector_rot1': -0.007522050071166131,
 'detector_rot2': -0.004845626736941386,
 'detector_rot3': 5.799041608456517e-08}

Tip

You can also use the param_values property to retrieve all parameter values in a dictionary.

4. Set the value of the xray_energy Parameter. This is a float value, however, the Parameter will attempt to convert other types to float. If successful, for demonstration purposes, let us set it with a string first. This will raise a :py:data:ValueError` and the Parameter will not be updated.

>>> exp.get_param_value('xray_energy')
15.0
>>> exp.set_param_value('xray_energy', '12.0')
>>> exp.get_param_value('xray_energy')
12.0
>>> exp.set_param_value('xray_energy', 13)
>>> exp.get_param_value('xray_energy')
13.0
>>> exp.set_param_value('xray_energy', "twelve")
ValueError: Cannot set Parameter (object ID:2129071369296, refkey:
'xray_energy', name: 'X-ray energy') because it is of the wrong data type.
(expected: <class 'numbers.Real'>, input type: <class 'str'>
>>> exp.get_param_value('xray_energy')
13.0

Global pydidas objects#

Global objects for processing#

All apps use the same global persistent objects (implemented as singletons), if required. Information is separated according to the reasons to change. The three main objects are:

ScanContext

The details about the scan. This includes generic information like scan title, data directory and scan names and specific information like the number of scan dimensions and the number of points in each dimension (but also metadata like dimension names, units, offsets and step width). The latter information can be used to create the correct axis labels in plots. For the full documentation please visit the ScanSetup manual.

DiffractionExperimentContext

This object includes information about the global experimental setup like X-ray energy, detector type, position and geometry. For the full documentation please visit the DiffractionExperimentContext manual.

WorkflowTree

The WorkflowTree holds information about which plugins are used and about the order of plugins to be processed. For the full documentation please visit the WorkflowTree manual.

These objects can be accesses by calling their respective factories:

>>> import pydidas
>>> SCAN = pydidas.contexts.ScanContext()
>>> EXPERIMENT = pydidas.contexts.DiffractionExperimentContext()
>>> TREE = pydidas.workflow.WorkflowTree()

Note that the factories return a link to the unique instance and multiple calls yield the same object:

>>> import pydidas
>>> SCAN = pydidas.contexts.ScanContext()
>>> SCAN
<pydidas.contexts.scan.Scan at 0x1d4a257b820>
>>> SCAN2 = pydidas.contexts.ScanContext()
>>> SCAN2
<pydidas.contexts.scan.Scan at 0x1d4a257b820>
>>> SCAN == SCAN2
True

PluginCollection#

pydidas uses a global PluginCollection to manage all known plugins. Plugins will be discovered based on known plugin paths which are managed persistently in the PluginCollection using Qt’s QSettings which use the systems registry and are platform-independent. A reference to the persistent PluginCollection object can be obtained using:

>>> import pydidas
>>> COLLECTION = pydidas.plugins.PluginCollection()

Note

For the full documentation of all available methods, please refer to the class documentation: PluginCollection This section handles only the most common use cases.

Management of stored paths#

Paths can be managed by various methods. New paths can be added using the find_and_register_plugins method and a list of all currently registered paths can be obtained by the get_all_registered_paths method. To permanently remove a stored paths, a user can use the unregister_plugin_path method. To remove all stored paths and plugin, use the unregister_all_paths method. This method must be called with a True flag to take effect and is ignored otherwise.

Note

Note that the generic plugin path is always included and cannot be removed from the collection by the user.

Warning

Using the unregister_all_paths method will remove all custom paths which have ever been registered and the user is responsible to add all required paths again.

This method will not keep any references to the original stored paths and they are truly lost.

An example of the use of stored paths is given below.

>>> import pydidas
>>> COLLECTION = pydidas.plugins.PluginCollection()
>>> COLLECTION.registered_paths
[Path(''/home/someuser/path/to/plugins')]
>>> COLLECTION.find_and_register_plugins(
    '/home/someuser/another/path',
    'home/someuser/yet/another/path')
>>> COLLECTION.registered_paths
[Path('/home/someuser/path/to/plugins'),
 Path('/home/someuser/another/path'),
 Path('/home/someuser/yet/another/path')]

# Now, if we exit and restart python, all paths will be included in the
# new instance:
>>> exit()
$ python
>>> import pydidas
>>> COLLECTION = pydidas.plugins.PluginCollection()
>>> COLLECTION.registered_paths
[Path('/home/someuser/path/to/plugins'),
 Path('/home/someuser/another/path'),
 Path(''/home/someuser/yet/another/path')]
# Using the ``unregister_plugin_path`` method allows to remove a single path:
>>> COLLECTION.unregister_plugin_path(Path('/home/someuser/another/path'))
>>> COLLECTION.registered_paths
[Path('/home/someuser/path/to/plugins'),
 Path(''/home/someuser/yet/another/path')]
# Using the ``unregister_all_paths`` method without the confirmation flag
# will be ignored:
>>> COLLECTION.unregister_all_paths()
'Confirmation for unregistering all paths was not given. Aborting...'
>>> COLLECTION.registered_paths
[Path('/home/someuser/path/to/plugins'),
 Path('/home/someuser/yet/another/path')]
>>> COLLECTION.unregister_all_paths(True)
>>> COLLECTION.registered_paths
[]

Plugin references#

Internally, plugins are referenced by their class name and there can only be one plugin registered with the same class name. This behaviour is deliberate to allow overwriting generic plugins with modified private versions. By default, plugin references are overridden with a new class if such a class is encountered. In addition to the class name, each plugin has a plugin_name attribute which allows to set a more readable reference name for the Plugin.

Tip

The loading of Plugins occurs in the order of the stored paths. Therefore, a path further down in the list will take precedence over an earlier path. The loading of Plugins can be controlled by organizing the sequence of paths.

The generic plugins are always loaded first.

Warning

Trying to register a second class with a different class name but the same plugin name will fail and raise an exception. Both the class name and the plugin name must be unique and a plugin can only replace a plugin with both matching class and plugin names or with a similar class name and a different plugin name.

Finding and getting a plugin#

Plugins can either be found by their class name using the get_plugin_by_name method or by their plugin name using the get_plugin_by_plugin_name method. A list of all available plugin class names can be obtained with the get_all_plugin_names method.

>>> import pydidas
>>> COLLECTION = pydidas.plugins.PluginCollection()
>>> COLLECTION.get_all_plugin_names()
['Hdf5fileSeriesLoader',
 'FrameLoader',
 'MaskImage',
 'PyFAI2dIntegration',
 'PyFAIazimuthalIntegration',
 'PyFAIradialIntegration',
 'pyFAIintegrationBase',
 'BasePlugin',
 'InputPlugin',
 'OutputPlugin',
 'ProcPlugin']

# Get the plugin class from the collection:
>>> _plugin = COLLECTION.get_plugin_by_name('PyFAI2dIntegration')
>>> _plugin
proc_plugins.pyfai_2d_integration.PyFAI2dIntegration

# Create a new instance:
>>> _integrate2d = _plugin()
>>> _integrate2d
<proc_plugins.pyfai_2d_integration.PyFAI2dIntegration at 0x2132e91a670>

# Get an azimuthal integration plugin by its plugin name and create a
# new instance directly (note the additional "()" at the end)
>>> _azi_int = COLLECTION.get_plugin_by_plugin_name('pyFAI azimuthal Integration')()
>>> _azi_int
<proc_plugins.pyfai_azimuthal_integration.PyFAIazimuthalIntegration at 0x2132e9b6ee0>

Once the plugins have been created, their Parameters can be modified as described in the Accessing Parameters section. The organization of plugins into a WorkflowTree are covered in the section WorkflowTree manual.

pydidas QSettings#

pydidas uses Qt’s QSettings to store persistent information in the system’s registry. The pydidas.core.PydidasQsettings class can be used to display and modify global parameters. The most useful methods for general users are show_all_stored_q_settings to print the names and values of all stored settings and set_value to modify a key.

>>> import pydidas
>>> config = pydidas.core.PydidasQsettings()
>>> config.show_all_stored_q_settings()
global/mp_n_workers: 4
global/plot_update_time: 1
global/shared_buffer_max_n: 20
global/shared_buffer_size: 100
>>> config.set_value('global/shared_buffer_size', 50)
>>> config.show_all_stored_q_settings()
global/mp_n_workers: 4
global/plot_update_time: 1
global/shared_buffer_max_n: 20
global/shared_buffer_size: 50

Note that the full list of global keys is longer and only a subset is presented here for demonstration purposes.

Note

The Qsettings are persistent (for a specific pydidas version) on the system for every individual user account, i.e. any changes you make will persist if you start a new pydidas instance or process. Likewise, any changes made as a different user will not be applied to your settings.

Description of pydidas’s Qsettings#

Pydidas QSettings are grouped into two main categories: ‘global’ settings and ‘user’ settings. Global settings define the system behaviour and performance whereas the user settings handle convenience user configurations.

Global settings#

  • Multiprocessing settings
    • Number of MP workers (key: global/mp_n_workers, type: int, default: 4)

      The number of multiprocessing workers. Note that this number should not be set too high for two reasons: 1. File reading processes interfere with each other if too many are active at once. 2. pyFAI already inherently uses parallelization and you can only gain limited performance increases for multiple parallel processes.

    • Shared buffer size limit (key: global/shared_buffer_size, type: float, default: 100, unit: MB)

      A shared buffer is used to efficiently transport data between the main App and multiprocessing Processes. This buffer must be large enough to store at least one instance of all result data.

  • Memory settings
    • Buffer dataframe limit (key: global/shared_buffer_max_n, type: int, default: 20)

      The maximum number of datasets in the buffer. A dataset consists of all results for one frame. For performance reasons, the buffer should not be too large as this is only a temporary buffer.

    • Maximum image size (key: global/max_image_size, type: float, default: 100, unit: MPixel)

      The maximum image size determines the maximum size of images pydidas will handle. The default is 100 Megapixels.

  • GUI plot update settings
    • Plot update time (key: global/plot_update_time, type: float, default: 1.0)

      The delay before any plot updates will be processed. This will prevent multiple frequent update of plots.)- Composite creation settings

User settings#

  • Update settings
    • Plot update time (key: user/auto_check_for_updates, type: bool, default: True)

      Flag to set the update check behaviour. By default, the GUI will check for a new version after starting up, if an active internet connection exists. This behaviour can be disabled by setting this flag to False.

  • Composite creator settings
    • Mosaic tiling border width (key: user/mosaic_border_width, type: int, default: 0, unit: px)

      The width of the border inserted between adjacent frames in the composite creation.

    • Mosaic border value (key: user/mosaic_border_value, type: float, default: 0)

      The value to be put in the border pixels in mosaics.

  • Plot settings
    • Histogram lower outlier fraction (key: user/histogram_outlier_fraction_low, type: float, default: 0.02)

      The lower fraction of the histogram to be cropped when using the “Crop histogram outliers” button in plots in the GUI. The value is the absolute fraction, i.e. the default of 0.02 will ignore the lowest 2% of pixels when setting the colormap from the histogram.

    • Histogram high outlier fraction (key: user/histogram_outlier_fraction_high, type: float, default: 0.07)

      The high fraction of the histogram to be cropped when using the “Crop histogram outliers” button in plots in the GUI. The value is the absolute fraction, i.e. the default of 0.07 will ignore the brightest 7% of pixels when setting the colormap from the histogram. The default of 0.07 is set to allow masking all the gaps in Eiger detectors by default.

    • Default colormap (key: user/cmap_name, type: str, default: Gray)

      The name of the default colormap to use. The name must be a valid matplotlib colormap name.

    • Default color for invalid data (key: user/cmap_nan_color, type: str, default: #9AFEFF)

      The RGB color code to use for invalid / missing data to distinguish it from the generic colormap and its data points.

  • Plugins
    • Custom plugin paths (key: user/plugin_path, type: str, default: “”)

      The names of custom paths where pydidas will look for plugins. Generic plugins and the generic plugin path is handled independently for greater clarity. Multiple paths can be added by appending them with a double semicolon “;;” as separator.

On this page
Edit on GitHub
Show Source