Running pydidas applications#

Pydidas applications are designed to be runnable both in serial and parallel modes. This requires a different syntax from the user which will be presented below.

Note

For a detailed description of the internal workings of applications please refer to the Developers guide to pydidas applications section.

Configuring an application#

pydidas applications are standard pydidas objects and their configuration is handled using pydidas Parameters. A short usage guide has already been given in Accessing Parameters.

Some applications might have additional properties or methods to further interact with the user, but these will be covered in the specific manuals.

As an example, let us use a DummyApp which creates a list with random strings and takes two parameters for the length of each string and the total number of strings.

Warning

The DummyApp is not a real object in pydidas. This example cannot be run in a real python console.

>>> import pydidas
>>> app = pydidas.apps.DummyApp()
>>> app.get_param_keys()
['number_of_strings', 'length_of_string']
>>> app.set_param_value('number_of_strings', 10)
>>> app.set_param_value('length_of_string', 10)

Running an application serially#

To run an app serially, simply call the run method of the app. Note that this will be a blocking call until the application is finished with its processing.

>>> import pydidas
>>> app = pydidas.apps.DummyApp()
>>> app.set_param_value('number_of_strings', 10)
>>> app.set_param_value('length_of_string', 10)
>>> app.run()
>>> app.get_results()
['yeOhsEodCq',
 'gnqZKRxHcK',
 'SbEbVlyjgx',
 'gcrNYzGUQY',
 'NnfLeTcXbS',
 'xoweEcFgqs',
 'ujHXTPsWyh',
 'XoOkaRqvIv',
 'ewrMXWBpdG',
 'TurPkkywwJ']

Running an application using parallelization#

Note

For a detailed description of how the pydidas multiprocessing works, please refer to the The multiprocessing sub-package or to the Developers guide to pydidas multiprocessing.

To run an application using parallel processing, an additional object is needed to control the parallelization. This is the AppRunner. Usage is straightforward and happens through only a few commands. The AppRunner starts in a separate thread and is non-blocking. It launches multiple independent processes and can run in the background.

To run an application, first configure the application as usually. Then, create an AppRunner instance with the configured application as calling argument. The app instance in the AppRunner is not directly accessible but the user can use the runner’s call_app_method method to call a method of the app or the set_app_param method to modify one of the application’s parameters.

Warning

Starting the AppRunner will create a new instance of the application and any changes made to the local instance will not be mirrored in the AppRunner’s app instance.

Running an application with the :py:class`AppRunner <pydidas.multiprocessing.AppRunner>` requires to call the start method. This will start the thread and create the worker processes.

Warning

Running apps which rely on Qt’s Signals and Slots (like the ExecuteWorkflowApp) will require a running QApplication event loop. Also do not forget to stop the event loop when finished.

An example with a custom application is given below:

import random
import string

import numpy as np
from qtpy import QtWidgets

import pydidas


#define CHARS to create random strings
CHARS = string.ascii_letters + string.digits


class DummyApp(pydidas.core.BaseApp):
    default_params = pydidas.core.ParameterCollection(
        pydidas.core.Parameter('number_of_strings', int, 10),
        pydidas.core.Parameter('length_of_strings', int, 10),
    )

    def multiprocessing_pre_run(self):
        self.results = {}
        pydidas.core.BaseApp.multiprocessing_pre_run(self)

    def multiprocessing_get_tasks(self):
        """Get the dummy tasks."""
        return np.arange(self.get_param_value('number_of_strings'))

    def multiprocessing_func(self, index):
        """Create a random string."""
        length = self.get_param_value('length_of_strings', 10)
        return ''.join(random.choice(CHARS) for _ in range(length))

    def multiprocessing_store_results(self, index, result):
        self.results[index] = result


def main():
    qtapp = QtWidgets.QApplication.instance()
    app = DummyApp()
    runner = pydidas.multiprocessing.AppRunner(app)

    # Now connect the AppRunner's emitted results to our local app:
    runner.sig_results.connect(app.multiprocessing_store_results)

    # Also make sure to terminate the QApplication event loop after
    # finishing the calculations:
    runner.finished.connect(qtapp.quit)

    # Start the runner and the QApplication event loop:
    runner.start()
    qtapp.exec_()

    print('Resulting random strings:')
    for _key, _val in app.results.items():
        print(f'  {_key:02d}: {_val}')


if __name__ == '__main__':
    main()

Running this script will create an output like:

Resulting random strings:
EgLDoQCBto
tbE07t910T
JpmZZJ6QjL
YTgKKtLWqP
vvFAuJ8W9d
HFLVkWt1Gm
kPjP2pWA6Z
AHiPGRADWa
1vFmmMM2mZ
BSrpoTxzOg

For status updates, one could use the AppRunner.sig_process Signal. An example with a new printing slot and an updated main function is given below:

def print_progress(progress):
    """Print the current progress repeatedly in the same line."""
    _n_chars = int(60 * progress)
    _txt = (
        "\u2588" * _n_chars
        + "-" * (60 - _n_chars)
        + "|"
        + f" {100*progress:05.2f}%: "
    )
    print(_txt, end='\r', flush=True)

def main():
    qtapp = QtWidgets.QApplication.instance()
    app = DummyApp()
    runner = pydidas.multiprocessing.AppRunner(app)

    # Now connect the AppRunner's emitted results to our local app:
    runner.sig_results.connect(app.multiprocessing_store_results)

    # Also make sure to terminate the QApplication event loop after
    # finishing the calculations:
    runner.finished.connect(qtapp.quit)

    # Connect also the AppRunner.sig_progress to the print_progress
    # function:
    runner.sig_progress.connect(print_progress)

    # Start the runner and the QApplication event loop:
    runner.start()
    qtapp.exec_()

    print('\nResulting random strings:')
    for _key, _val in app.results.items():
        print(f'  {_key:02d}: {_val}')