MSO4SC: e-Infrastructure requirements

Requirements describe the desired features and functionality of a product, here the MSO4SC e-infrastructure. A requirement is something that the e-infrastructure must do or a quality that is must have (it is also possible to consider constraints); either the e-infrastructure demands certain functionalities or qualities, or the stakeholders (clients) want that requirement to be part of the e-infrastructure. Requirements are divided into two classes. A functional requirement is something that the e-infrastructure must do if it is to be useful within its intended context. Non-functional requirements are properties, or qualities, that the e-infrastructure must have. An important example for a non-functional requirement in the context of the intended MSO4SC e-infrastructure with its mathematical application frameworks are performance requirements (how fast, big, accurate, safe, reliable, scalable).

Requirements list

1. Cloud-1: Generic application support

Table 1. Requirement table Cloud-1 Generic application support

# Id

Cloud-1

Name

Generic application support

Priority

Medium

Req. Type

Functional

Description

The infrastructure should provide support to other MADFs and applications not described in the project. This means that it should be flexible and generic enough that it will not stick to a concrete domain or set of tools.

Purpose

For the sustainability of the project the infrastructure should have the availability to provide support to other user communities beyond those which are part of the consortium.

Actors

CESGA, ATOS

Requester

Stakeholders

Validation scenario

New stakeholders will be invited to use the MSO4SC which should be able to run the new simulations. At least one external application will be executed correctly.

KPIs: Number of applications tested on MSO4SC e-Infrastructure, Number of executions performed per application, Singularity versions tested, Singularity containers tested, Success/failure ratio, identified issues.

Implementation and Validation

Implementation: 100%;

Validation: 90% (some issues found, to be fixed)

Components

Orchestrator, Monitor, MSO portal

Relationships

NA

2. Cloud-2: Multiple infrastructure support

Table 2. Requirement table Cloud-2 Multiple infrastructure support

# Id

Cloud-2

Name

Multiple infrastructures support

Priority

High

Req. Type

Functional

Description

The software developed to manage and orchestrate the infrastructure should be portable and capable to manage other infrastructures, not only the ones and the beginning of the project (CESGA & ATOS).

Purpose

Some use cases are very restricted in terms of data confidentiality, so some parts of the software have to run in local clusters of the users. This should be supported.

Actors

ATOS, CESGA, SINTEF

Requester

SINTEF (OPM Flow use case)

Validation scenario

The OPM Flow use case will be able to execute part of the scenario in an infrastructure different from the ones provided by CESGA and ATOS.

KPIs: Support for federated HPC/Cloud, Types of supported infrastructures, Types of supported resource managers, Number of tests.

Implementation and Validation

Implementation: 100%;

Validation: 100%

Components

Orchestrator

Relationships

NA

3. Cloud-3: Data management

Table 3. Requirement table Cloud-3 Data management

# Id

Cloud-3

Name

Data management

Priority

High

Req. Type

Functional

Description

The software developed to manage and orchestrate the infrastructure should be able to move data among different infrastructures in a secure way, especially when different parts of the application are running in different hardware infrastructures.

Purpose

This functionality is needed to provide the data for the simulations and the output results in the different places where the simulations might take place

Actors

CESGA, ATOS, SZE

Requester

SZE (3DAirqualityprediction use case), survey

Validation scenario

In the 3DAirqualityprediction use case, the application will be split in two hardware infrastructures for its execution, and all the data will be moved correctly from one infrastructure to the other one, as required by the application components.

KPIs: Number of tests, Number of infrastructures involved, Transfer rates, Application start time.

Implementation and Validation

Implementation: 100%;

Validation: 90% (transfer rates reached not always as expected, but good enough in general)

Components

Orchestrator

Relationships

Cloud-2

4. Cloud-4: Licenses and software restrictions

Table 4. Requirement table Cloud-4 Licenses and software restrictions

# Id

Cloud-4

Name

Cloud-4

Priority

Medium

Req. Type

Medium

Description

The software developed to manage and orchestrate the infrastructure should support proprietary software and the restrictions that it might impose.

Purpose

Some use cases make use of proprietary software. This software is not installed in all the clusters because of license restrictions so the orchestration must take this in account when deploying and running the simulations

Actors

CESGA, ATOS, SZE

Requester

SZE (3DAirqualityprediction use case) and UNISTRA (Eye2Brain and HIFIMAGNET uses cases)

Validation scenario

During the 3DAirqualityprediction, Exe2Brain and HIFIMAGNET use cases, at least one of the components deployed will be proprietary and the MSO4SC e-Infrastructure will be able to deploy and execute it without any issues related to the licensing management.

KPIs: Number of licenses managed, Types of licenses, Number of deployed applications using licenses

Implementation and Validation

Implementation: 100%;

Validation: 100%

Components

Orchestrator

Relationships

NA

5. Cloud-5: Support MPI Applications

Table 5. Requirement table Cloud-5 Support MPI Applications

# Id

Cloud-5

Name

Support MPI Applications

Priority

High

Req. Type

Functional

Description

The software developed to manage and orchestrate the infrastructure should support proprietary software and the restrictions that it might impose.

Purpose

Some use cases make use of proprietary software. This software is not installed in all the clusters because of license restrictions so the orchestration must take this in account when deploying and running the simulations

Actors

CESGA, ATOS, SZE

Requester

SZE (3DAirqualityprediction use case) and UNISTRA (Eye2Brain and HIFIMAGNET uses cases)

Validation scenario

During the 3DAirqualityprediction, Exe2Brain and HIFIMAGNET use cases, at least one of the components deployed will be proprietary and the MSO4SC e-Infrastructure will be able to deploy and execute it without any issues related to the licensing management.

KPIs: Number of MADFs deployed, Number of pilots deployed, Number of times the MADFs and pilots were run, Number of failures due to issues in the installation, Number of MPI libraries tested, Number of nodes used.

Implementation and Validation

Implementation: 100%;

Validation: 100%

Components

Orchestrator

Relationships

NA

6. Cloud-6: Visualization

Table 6. Requirement table Cloud-6 Visualization

# Id

Cloud-6

Name

Visualization

Priority

High

Req. Type

Functional

Description

The infrastructure should support remote visualization capabilities, so it will be possible to load the results of a simulation and watch them. The infrastructure should support an integrated use of visualization software as part of the services to the users.

Purpose

Many use cases require visualization software in different parts of the simulation: pre-processing and post-processing, mainly.

Actors

CESGA, ATOS, UNISTRA

Requester

ZIBaffinity users, ZIB, survey

Validation scenario

At least 4 pilots will use visualization tools in order to show the simulation results. Such visualization should be smoothly integrated with the rest of functionalities in the MSO Portal.

KPIs: Visualization techniques/alternatives, MADFS and Pilots requiring visualization, Formats supported, Number of performed tests, Visualization software, Success/failure ratio.

Implementation and Validation

Implementation: 100%;

Validation: 100%

Components

MSO Portal

Relationships

NA

7. Cloud-7: Simulation monitoring

Table 7. Requirement table Cloud-7 Simulation monitoring

# Id

Cloud-7

Name

Simulation monitoring

Priority

Medium

Req. Type

Functional

Description

The infrastructure should support remote visualization capabilities, so it will be possible to load the results of a simulation and watch them. The infrastructure should support an integrated use of visualization software as part of the services to the users.

Purpose

Many use cases require visualization software in different parts of the simulation: pre-processing and post-processing, mainly.

Actors

CESGA, ATOS, UNISTRA

Requester

ZIBaffinity users, ZIB, survey

Validation scenario

At least 4 pilots will use visualization tools in order to show the simulation results. Such visualization should be smoothly integrated with the rest of functionalities in the MSO Portal.

KPIs: Number of infrastructures, Number of applications, Number of application metrics shown, Number of infrastructure metrics shown, Number of tests.

Implementation and Validation

Implementation: 100%;

Validation: 100%

Components

MSO Portal

Relationships

NA

8. Cloud-8: Automate applications deployment and execution

Table 8. Requirement table Cloud-8 Automate applications deployment and execution

# Id

Cloud-8

Name

Automate applications deployment and execution

Priority

High

Req. Type

Functional

Description

The MSO4SC e-Infrastructure will provide a rich GUI in order to submit jobs to HPC systems and tasks to Cloud providers. A process in the background will be responsible to prepare the application, deploy it and run it in the corresponding context (HPC and/or Cloud) without the need of user intervention in the technical aspects.

Purpose

MSO4SC stakeholders do not need to be aware of the technical details when asking for the execution of an application. Therefore, MSO4SC e-Infrastructure should ease such task by abstracting the technical complexity, with the proper mechanisms running in the background.

Actors

ATOS, CESGA

Requester

MSO4SC partners, Stakeholders

Validation scenario

All the project pilots will evaluate the functionality. Stakeholders will run the applications through a GUI from the MSO Portal, and the right deployment and execution will be done by MSO4SC in 100% of the cases (without any error related to the orchestration mechanism itself).

KPIs: Number of buttons to click to execute a simulation.

Implementation and Validation

Implementation: 100%;

Validation: 100%

Components

Orchestrator, MSO Portal

Relationships

NA

9. Cloud-9: Decoupling of simulations and visualization

Table 9. Requirements table Cloud-9 Decoupling of simulations and visualization

# Id

Cloud-9

Name

Decoupling of simulations and visualization

Priority

Medium

Req. Type

Non-Functional

Description

Simulations run in MSO4SC should provide the possibility to use any tool for visualization, instead of a concrete one for all the cases. This means that it will be possible to generate the simulation outcome and just download it if desired, so the stakeholders will be able to select the visualization tool they want and provide such outcome as input. All the MADFs will store simulation outcomes in a location from which the MSO Portal will be able to access to serve it to stakeholders.

Purpose

MSO4SC stakeholders want to avoid vendor lock-in in the case of visualization, since each one may have their own preferences in term of post-processing and visualization tools.

Actors

ATOS, CESGA, KTH, BCAM, UNISTRA, SINTEF

Requester

Stakeholders

Validation scenario

Several simulations will generate outcomes that can be retrieved directly by stakeholders. Such functionality will be tested separately with all the pilots, checking that in 100% of the cases it is possible to retrieve the simulation result, and use it in the desired visualization tool (after some post-processing, if required).

KPIs: Visualization alternatives, Visualization formats supported.

Implementation and Validation

Implementation: 100%;

Validation: 100%

10. Cloud-10: Optimize data management

Table 10. Requirements table Cloud-10 Optimize data management

# Id

Cloud-10

Name

Optimize Data Management

Priority

High

Req. Type

Non Functional

Description

The software developed to manage and orchestrate the infrastructure should be able to move data among different infrastructures, some of them dedicated to particular tasks so that optimise data movement times.

Purpose

This functionality is needed to ensure movement of measured data of the end-user application represented on dedicated servers to the MSO4SC infrastructure while keeping good overall execution times.

Actors

SZE, CESGA, ATOS

Requester

external parties, SZE

Validation scenario

One of the pilots should be able to show that data movement has been optimized and simplified.

KPIs: Can be automated, Supported storage endpoints, Supported data movers.

Implementation and Validation

Implementation: 100%;

Validation: 100%

Components

Orchestrator

Relationships

3DAQP-2

11. Cloud-11: Pause/Restart/Recover simulation

Table 11. Requirements table Cloud-11 Pause/Restart/Recover simulation

# Id

Cloud-11

Name

Pause/Restart/Recover simulation

Priority

High

Req. Type

Functional

Description

The users request a way to pause or stop a simulation in order to reconfigure it and start again, from the beginning or a certain point, from an adequate GUI.

Purpose

In some simulations, it is very useful to recalibrate the simulation if something is wrong or could be improved, and restart it without starting over again, especially if the resources consumed during the simulation are high.

Actors

MADF owners, ATOS

Requester

Stakeholders, survey

Validation scenario

A modularized simulation using one of the MADFs will be stopped, reconfigured and restarted from the same point, and the results obtained will be the expected ones.

KPIs: Number of applications requiring it, Number of tests.

Implementation and Validation

Implementation: 70%;

Validation: 70%

Components

MADFs, MSO Portal

Relationships

Cloud-7

12. Cloud-12: Community Support

Table 12. Requirements table Cloud-12 Community Support

# Id

Cloud-12

Name

Community Support

Priority

Medium

Req. Type

Functional

Description

Different mechanisms should be provided to allow users an easy learning curve of MSO4SC, such as Q&A, Wikis or forums. These mechanisms will be the way to articulate the community around MSO4SC.

Purpose

Stakeholders (members and no-members of the project) will have questions about how the different parts of MSO4SC work, as well as need some kind of training to start using it quickly. Such functionality will allow a higher level of software maturity (TRL).

Actors

ATOS, CESGA

Requester

MSO4SC partners

Validation scenario

Users will be able to post/answer more than 3 questions, read and edit the wikis, and participate in, at least one learning event. The MSO Portal will reach, at least, TRL 8.

KPIs: Number of documented services, Support platforms, Support plan.

Implementation and Validation

Implementation: 100%;

Validation: 100%

Components

MSO Portal

Relationships

NA

13. Cloud-13: Data categorization

Table 13. Requirements table Cloud-13 Data categorization

# Id

Cloud-13

Name

Data categorization

Priority

High

Req. Type

Functional

Description

The data management system should provide a way to easily find the different datasets available and generated in the infrastructure. That means that there should be available metadata about the datasets and that a search mechanism will be in place.

Purpose

MSO4SC users need to navigate through the data catalogue available in the portal, quickly finding the dataset they need.

Actors

ATOS

Requester

MSO4SC Partners, survey

Validation scenario

Concrete datasets are found in only one query.

KPIs: Number of predefined labels, Number of custom labels used, Number of datasets, Number of pilots with published datasets.

Implementation and Validation

Implementation: 100%;

Validation: 100%

Components

MSO Portal

Relationships

NA

14. Cloud-14: Multi-Simulation

Table 14. Requirements table Cloud-14 Multi-Simulation

# Id

Cloud-14

Name

Multi-Simulation

Priority

Medium

Req. Type

Functional

Description

The MSO4SC e-Infrastructure should enable the possibility to run embarrassingly parallel tasks, in which multiple simulations need to be launched (sometimes varying a few parameters) in such a way they are related.

Purpose

MSO4SC users may need to execute ensembled simulations (such as in the case of OPM).

Actors

ATOS

Requester

MSO4SC Partners, survey

Validation scenario

It is possible to run several ensembled simulations successfully, in the context of one of the pilots.

KPIs: Number of ensemble applications tested, Scalability and number of jobs per application, Number of executions per application.

Implementation and Validation

Implementation: 100%;

Validation: 100%

Components

MSO Portal

Relationships

NA

15. Cloud-15: Containers support

Table 15. Requirements table Cloud-15 Containers support

# Id

Cloud-15

Name

Containers support

Priority

High

Req. Type

Non-Functional

Description

The infrastructure must run HPC jobs inside containers with no significant latency.

Purpose

Containers will include the application to run and the right combination of libraries and versions, in such a way it will not be necessary to reconfigure the target hardware every time an application is deployed.

Actors

All MSO4SC Partners

Requester

MSO4SC Partners

Validation scenario

Stakeholders providing its simulations with all their dependencies packed in a container should be able to execute them without perceiving delays. There will be containers for all the pilots and the MADFs, and performance loss should be below 5%.

KPIs: Number of containerized applications, Number of execution of containerized applications, Number of containerized benchmarks, Singularity versions tested.

Implementation and Validation

Implementation: 100%;

Validation: 100%

Components

Orchestrator, MADFs, Pilots

Relationships

NA

16. Cloud-16: Open development

Table 16. Requirements table Cloud-16 Open development

# Id

Cloud-16

Name

Open development

Priority

High

Req. Type

Non-Functional

Description

The infrastructure source code and the evolution of its development should be open and hosted in common public repositories (e.g. github).

Purpose

Internal and external developers can collaborate and help to improve, maintain and fix the E-Infrastructure itself.

Actors

MSO4SC Partners and external developers

Requester

MSO4SC Partners

Validation scenario

Developers community can access to the source code, submit improvements and issues.

KPIs: Number of external contributions, Number of external projects using parts of MSO4SC e-Infrastructure, Number of issues published by external users.

Implementation and Validation

Implementation: 100%;

Validation: 100%

Components

Portal, orchestrator, monitor

Relationships

NA

17. Cloud-17: Private data and security

Table 17. Requirement table Cloud-17 Private data and security

# Id

Cloud-17

Name

Private data and security

Priority

High

Req. Type

Functional

Description

Users should require to work with private or confidential data for both inputs and outputs. Users can decide the visibility of the input and output data, and the e-infrastructure must ensure the required security level for these data.

Purpose

The end user need the above feature to manage permissions on the inputs and outputs generated by the simulations to get control on the potential credentials or confidentials issues.

Actors

CESGA, ATOS, SZE, End-users, survey

Requester

All use cases

Validation scenario

If the end-user does not explicitly share any particular owned file or directory, these files are only accessible for him or her. If the end user share a file or folder them will be publicly available for all users.

KPIs: Number of private datasets, Number of organization with private datasets.

Implementation and Validation

Implementation: 100%;

Validation: 100%

Components

Cloud, Data repository

Relationships

Cloud-3, Cloud-10, Cloud-13

18. Cloud-18: Pre- and post-processing

Table 18. Requirement table Cloud-18 Pre- and post-processing

# Id

Cloud-18

Name

Pre- and postprocessing

Priority

High

Req. Type

Functional

Description

The infrastructure should support remote as well as integrated pre- and postprocessing capabilities as part of the services to the users.

Purpose

Many use cases require pre- and/or postprocessing for their simulation.

Actors

CESGA, ATOS

Requester

FEniCS users, survey

Validation scenario

At least 1 pilot will use pre- and/or postprocessing tools for their simulation results.

KPIs: Number of pre- and postprocess tool, Number of pilots using pre- and/or postprocess tools, Number of tests with pre- and/or postprocess tools.

Implementation and Validation

Implementation: 100%;

Validation: 100%

Components

MSO Portal

Relationships

NA

19. Cloud-19: Online editing for inputs

Table 19. Requirement table Cloud-19 Online editing for inputs

# Id

Cloud-19

Name

Online editing for inputs

Priority

High

Req. Type

Functional

Description

The portal should allow the possibility to edit the inputs as a list of text lines, in such a way that they can be used, later on, for creating several instances of the simulation with different inputs.The portal will allow to indicate that the edited file is the input for the simulation, retrieving the expected values. The initial content could be a template provided by the application developer.

Purpose

Stakeholders may want to adapt their mathematical models before running their simulations, or even just parametrize them with different inputs listed in a file.

Actors

CESGA, ATOS

Requester

FEniCS users, maths community

Validation scenario

At least 1 pilot will modify a text with the desired inputs and select it as input for a simulation, showing that a new instance is generated with the new configuration.

KPIs: Number applications using online editing, Number of files/text edited, number of runs with edited inputs.

Implementation and Validation

Implementation: 100%;

Validation: 100%

Components

MSO Portal

Relationships

NA

How to contribute

We use the following table to define requeriments.

Table 20. Requirements table template

# Id

ID

Name

Name for the requirement

Priority

Low/Medium/High

Req. Type

Functional/Non-functional

Description

Definition of the requirement. Describe what it is about.

Purpose

Reason to include the requirement. Justify why the requirement should be taken into account.

Actors

Actors involved in the requirement, taking into account stakeholders related to it. The ones who must deal with it.

Requester

List of pilots or entities which proposed or are related to the requirement and that can validate the requirement.

Validation scenario

Determine some validation criteria which would check that the requirement is fulfilled (i.e. small remark about the testing that should be done). List the metrics (KPIs) that should be measured.

Implementation and Validation

Determine percentage of the implementation of the requirements and the percentage of validation.

Components

Include some mapping with the parts of the high level architecture which are affected by this requirement

Relationships

List here those requirements related to this one

To suggest a new requirement for the e-Infrastructure, please, click on Edit this page (at the top right corner of this web page) and create a new pull request with the proposed requirement.

The asciidoc source code for a requirement template is shown below.

Requirement template in asciidoc
[cols=",,,",]
.Requirements table template
|==================================================================================================================================================
a| *# Id*
a| ID
a| *Name*
 | Name for the requirement
a| *Priority*
a| Low/Medium/High
a| *Req. Type*
 | Functional/Non-functional
a| *Description*
 | Definition of the requirement. Describe what it is about.
a| *Purpose*
 | Reason to include the requirement. Justify why the requirement should be taken into account.
a| *Actors*
 | Actors involved in the requirement, taking into account stakeholders related to it. The ones who must deal with it.
a| *Requester*
 | List of pilots or entities which proposed or are related to the requirement and that can validate the requirement.
a| *Validation scenario*
 | Determine some validation criteria which would check that the requirement is fulfilled (i.e. small remark about the testing that should be done). List concrete  metrics (KPIs) to measure.
a| *Implementation and Validation*
 | Percentage of implementation and validation of the requirements (0% initially).
a| *Components*
 | Include some mapping with the parts of the high level architecture which are affected by this requirement
a| *Relationships*