Alternative way for link passing and substitution in markdown
@ -46,11 +46,6 @@ Actual program: [SPC-A-6](https://npm.mipt.ru/youtrack/articles/SPC-A-6)
|
|||||||
|
|
||||||
Actual program: [SPC-A-3](https://npm.mipt.ru/youtrack/articles/SPC-A-3)
|
Actual program: [SPC-A-3](https://npm.mipt.ru/youtrack/articles/SPC-A-3)
|
||||||
|
|
||||||
### [Instruments of development](https://npm.mipt.ru/youtrack/articles/SPC-A-5)
|
|
||||||
*curated by [Alexander Nozik](${resolvePageRef("team.index")}#nozik)*
|
|
||||||
|
|
||||||
Actual program: [SPC-A-5](https://npm.mipt.ru/youtrack/articles/SPC-A-5)
|
|
||||||
|
|
||||||
### [Advanced Python](https://npm.mipt.ru/youtrack/articles/SPC-A-4)
|
### [Advanced Python](https://npm.mipt.ru/youtrack/articles/SPC-A-4)
|
||||||
*by Mikhail Zelenyy*
|
*by Mikhail Zelenyy*
|
||||||
|
|
||||||
|
109
data/home/content/projects/bmn.md
Normal file
@ -0,0 +1,109 @@
|
|||||||
|
---
|
||||||
|
type: project
|
||||||
|
title: BM@N infrastructure
|
||||||
|
order: 150
|
||||||
|
language: en
|
||||||
|
image: images/projects/bmn/nica.png
|
||||||
|
---
|
||||||
|
## Information Systems for BM@N
|
||||||
|
|
||||||
|
Our team works on development of information systems and services for BM@N (Baryonic Matter at Nuclotron) experiment, part of NICA (Nuclotron-based Ion Collider fAсility) megaproject (located in Dubna, Russia). These works are performed together with scientists from JINR and other institutions.
|
||||||
|
|
||||||
|
The overall view of NICA complex with already running experiment BM@N and future collider experiments MPD and SPD is shown in Fig. 1. BM@N studies collisions of elementary particles and ions with a fixed target with energies up to 6 GeV per nucleon, which is a very interesting energy region for fundamental nuclear physics research.
|
||||||
|
|
||||||
|
<span class="image fit">
|
||||||
|
<strong>Fig. 1</strong>
|
||||||
|
<img src="snark://ref/images/projects/bmn/nica.png" alt="Fig. 1"/>
|
||||||
|
</span>
|
||||||
|
|
||||||
|
### Event Metadata System
|
||||||
|
|
||||||
|
Event metadata systems (EMS) are widely used in the physics experiments on particle collisions, where large numbers of experimental events, typically billions, are collected. At the stage of particular physics analysis of experimental data, only a small subset of collected events meeting certain criteria is usually of interest, and passing over the whole amount of stored data to form the subset is overly time-consuming and resource-intensive. Instead, metadata systems allow one to obtain only the required events (or, at least, references to them) via searching and filtering based on given fields (event attributes) recorded in the metadata system.
|
||||||
|
|
||||||
|
The NICA Event Metadata System has been developed for storing necessary event metadata for the NICA experiments. The information system makes it possible to quickly search for a necessary subset of physics events by required parameters to use in further event data processing. It provides summary description of collision events and their identifiers to search and select events for a desired analysis task, enables their management and convenient access; provides online and offline interfaces for selecting events of interest, such as Web and REST API services, and a dedicated C++/ROOT interface.
|
||||||
|
|
||||||
|
The Event Metadata System, including the Event Catalogue based on PostgreSQL, Metadata API, Web Service, and other developed components, has been provisioned using a common deployment system for the first experiment of the NICA project, BM@N. The main interfaces to EMS, namely, Web UI and REST API have been developed using Kotlin multiplatform technology as a part of the single full-stack project. The EMS provides acceptable response times for the expected amount of the event metadata. The system and its services will be further evolved and improved following operational experience.
|
||||||
|
|
||||||
|
Figs. 2 and 3 illustrate high-level EMS architecture and its basic Web UI view.
|
||||||
|
|
||||||
|
<span class="image fit">
|
||||||
|
<strong>Fig. 2</strong>
|
||||||
|
<img src="snark://ref/images/projects/bmn/ems-arch.png" alt="Fig. 2"/>
|
||||||
|
</span>
|
||||||
|
|
||||||
|
<span class="image fit">
|
||||||
|
<strong>Fig. 3</strong>
|
||||||
|
<img src="snark://ref/images/projects/bmn/ems-web.png" alt="Fig. 3"/>
|
||||||
|
</span>
|
||||||
|
|
||||||
|
### Next-Generation Event Visualization (Event Display) System
|
||||||
|
|
||||||
|
In high-energy physics experiments, the ability to display both detector geometry and particle tracks has become an essential feature, required for physicists to better understand particular collision events as well as to present the physical results to a wider audience.
|
||||||
|
|
||||||
|
For BM@N experiment, a new event visualization solution was developed based on VisionForge, a modern open-source visualization system. An important part of the solution is integration of the system with experiment's software framework BmnRoot, which is a CERN ROOT-based environment.
|
||||||
|
|
||||||
|
|
||||||
|
Figs. 4, 5 illustrate how the Web interface of a new system looks like. The user can conveniently move and rotate the camera, zoom-in and zoom-out the scene as needed, browse the scene graph (object tree). For every object it is possible to change display properties such as color, opacity and visibility. Note that the full BM@N geometry model includes more than 400,000 primitives so rendering them all requires significant resources. To optimize the resource usage, so-called “prototypes” were implemented in VisionForge model for three-dimensional objects. For a single prototype its geometry is rendered only once and reused for multiple objects, which helps to significantly reduce memory usage.
|
||||||
|
|
||||||
|
<span class="image fit">
|
||||||
|
<strong>Fig. 4</strong>
|
||||||
|
<img src="snark://ref/images/projects/bmn/vis-1.png" alt="Fig. 4"/>
|
||||||
|
</span>
|
||||||
|
|
||||||
|
<span class="image fit">
|
||||||
|
<strong>Fig. 5</strong>
|
||||||
|
<img src="snark://ref/images/projects/bmn/vis-2.png" alt="Fig. 5"/>
|
||||||
|
</span>
|
||||||
|
|
||||||
|
### Condition Database Services
|
||||||
|
|
||||||
|
The Unified Condition Database (shortly called UniConDa) of the BM@N experiment is one of the advanced information systems required to support automation of experimental data collecting and processing, providing a central storage for experiment metadata being necessary for event data processing, including session and run information, detector and subsystem parameters, and descriptions of simulated event files.
|
||||||
|
|
||||||
|
The condition database has been earlier implemented based on the PostgreSQL database management system in accordance with a well-designed scheme. An ecosystem of its supplementary services is constantly being improved to increase overall degree of database integration and convenience of the use by collaboration members.
|
||||||
|
|
||||||
|
For the UniConDa ecosystem, the REST API, a modern HTTP-based interface for integration with both ROOT-based and non-ROOT software systems of the experiment, was developed.
|
||||||
|
|
||||||
|
Additionally, Smart Data Parser, a part of the BM@N information system based on the Unified Condition Database, was created to solve the task of the parameter data migration from accumulated files of ASCII text, CSV or XML formats to the Unified Condition Database of the BM@N experiment (Fig. 6).
|
||||||
|
|
||||||
|
<span class="image fit">
|
||||||
|
<strong>Fig. 6</strong>
|
||||||
|
<img src="snark://ref/images/projects/bmn/sdp.png" alt="Fig. 6"/>
|
||||||
|
</span>
|
||||||
|
|
||||||
|
### Monitoring Service
|
||||||
|
|
||||||
|
The software infrastructure of the BM@N experiment contains a set of various information systems that are essential for the work with experimental or simulated data on all processing stages, including the collection, storage, intermediate processing and physics analysis. In case one of such systems stops functioning, the work with BM@N data by collaboration members gets either impossible or, at least, much less productive. Due to this fact, the timely detection of possible failures in the systems due to software or hardware failures is fairly important.
|
||||||
|
|
||||||
|
The developed Monitoring Service is used to check availability and health status of information systems. This includes measuring, storing, visualizing and sending alert notifications on monitored parameters, such as CPU, memory and disk utilization, DBMS functioning parameters, response times of databases and API endpoints, ping round-trip times, and so on. The current implementation of the BM@N monitoring service is illustrated in Figs. 7, 8. Note that a related task of building highly available information services is currently a work in progress.
|
||||||
|
|
||||||
|
<span class="image fit">
|
||||||
|
<strong>Fig. 7</strong>
|
||||||
|
<img src="snark://ref/images/projects/bmn/mon-arch.png" alt="Fig. 7"/>
|
||||||
|
</span>
|
||||||
|
|
||||||
|
<span class="image fit">
|
||||||
|
<strong>Fig. 8</strong>
|
||||||
|
<img src="snark://ref/images/projects/bmn/mon-db.png" alt="Fig. 8"/>
|
||||||
|
</span>
|
||||||
|
|
||||||
|
|
||||||
|
### Slow Control System Viewer
|
||||||
|
|
||||||
|
Web interface for slow control system of BM@N was developed. It visualizes various sensor data graphs based on parameters and time intervals provided by user as shown in Fig. 9.
|
||||||
|
|
||||||
|
<span class="image fit">
|
||||||
|
<strong>Fig. 9</strong>
|
||||||
|
<img src="snark://ref/images/projects/bmn/scs.png" alt="Fig. 9"/>
|
||||||
|
</span>
|
||||||
|
|
||||||
|
### References
|
||||||
|
|
||||||
|
1. K. Gertsenberger, P. Klimai, M. Zelenyi. Auxiliary Services for the Condition Database of the BM@N Experiment at NICA // Phys.Part.Nucl.Lett. 20 (2023) 1217. https://www.doi.org/10.1134/S1547477123050291
|
||||||
|
2. E. Alexandrov, I. Alexandrov, A. Chebotov, A. Degtyarev, I. Filozova, K. Gertsenberger, P. Klimai, A. Yakovlev. Implementation of the Event Metadata System for physics analysis in the NICA experiments // Journal of Physics: Conference Series 2438 (2023) 012046. https://www.doi.org/10.1088/1742-6596/2438/1/012046
|
||||||
|
2. A. Degtyarev, K. Gertsenberger, P. Klimai. Usage of Apache Cassandra for Prototyping the Event Metadata System of the NICA Experiments // Phys.Part.Nucl.Lett. 19 (2022) 5, 562-565. https://doi.org/10.1134/S1547477122050144
|
||||||
|
3. A. Chebotov, K. Gertsenberger, P. Klimai, A. Moshkin. Information System Based on the Condition Database for the NICA Experiments, User WEB Application, and Related Services // Phys.Part.Nucl.Lett. 19 (2022) 5, 558-561. https://doi.org/10.1134/S1547477122050132
|
||||||
|
4. E. Alexandrov, I. Alexandrov, A. Degtyarev, K. Gertsenberger, I. Filozova, P. Klimai, A. Nozik, A. Yakovlev. Design of the Event Metadata System for the Experiments at NICA // Phys.Part.Nucl.Lett. 18 (2021) 5, 603-616. https://doi.org/10.1134/S1547477121050034
|
||||||
|
5. K. Gertsenberger, I. Alexandrov, I. Filozova, E. Alexandrov, A. Moshkin, A. Chebotov, M. Mineev, D. Pryahina, G. Shestakova, A. Yakovlev, A. Nozik, P. Klimai. Development of Information Systems for Online and Offline Data Processing in the NICA Experiments // Phys.Part.Nucl. 52 (2021) 4, 801-807. https://doi.org/10.1134/S1063779621040250
|
||||||
|
6. K.V. Gertsenberger, A.I. Chebotov, P.A. Klimai, I.N. Alexandrov, E.I. Alexandrov, I.A. Filozova, A.A. Moshkin. Implementation of the Condition Database for the Experiments of the NICA Complex // CEUR Workshop Proceedings, Vol. 3041 (2021) 128-132. https://doi.org/10.54546/MLIT.2021.53.54.001
|
||||||
|
7. E.I. Alexandrov, I.N. Alexandrov, A.G. Degtyarev, I.A. Filozova, K.V. Gertsenberger, P.A. Klimai, A.V. Yakovlev. Development of the Event Metadata System for the NICA experiments // CEUR Workshop Proceedings, Vol. 3041 (2021) 439-444. https://doi.org/10.54546/MLIT.2021.80.18.001
|
||||||
|
8. M.N.Kapishin et al, The Report on Project “Studies of Baryonic Matter at the Nuclotron (BM@N)”, https://bmn.jinr.ru/wp-content/uploads/2022/10/BMNproject_2021cor2.pdf
|
5
data/home/content/projects/bmn[info].md
Normal file
@ -0,0 +1,5 @@
|
|||||||
|
---
|
||||||
|
language: en
|
||||||
|
---
|
||||||
|
|
||||||
|
Our team works on development of information systems and services for BM@N (Baryonic Matter at Nuclotron) experiment, part of NICA (Nuclotron-based Ion Collider fAсility) megaproject (located in Dubna, Russia). These works are performed together with scientists from JINR and other institutions.
|
@ -21,7 +21,7 @@ A presentation on application of (old version of) DataForge to Troitsk nu-mass a
|
|||||||
<iframe width="100%" height="315" src="https://www.youtube.com/embed/OpWzLXUZnLI" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
|
<iframe width="100%" height="315" src="https://www.youtube.com/embed/OpWzLXUZnLI" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
|
||||||
|
|
||||||
## Questions and Answers
|
## Questions and Answers
|
||||||
In this section we will try to cover DataForge main ideas in the form of questions and answers.
|
In this section, we will try to cover DataForge main ideas in the form of questions and answers.
|
||||||
|
|
||||||
### General
|
### General
|
||||||
**Q**: I have a lot of data to analyze. The analysis process is complicated, requires a lot of stages and data flow is not always obvious. To top it the data size is huge, so I don't want to perform operation I don't need (calculate something I won't need or calculate something twice). And yes, I need it to be performed in parallel and probably on remote computer. By the way, I am sick and tired of scripts that modify other scripts that control scripts. Could you help me?
|
**Q**: I have a lot of data to analyze. The analysis process is complicated, requires a lot of stages and data flow is not always obvious. To top it the data size is huge, so I don't want to perform operation I don't need (calculate something I won't need or calculate something twice). And yes, I need it to be performed in parallel and probably on remote computer. By the way, I am sick and tired of scripts that modify other scripts that control scripts. Could you help me?
|
||||||
|
BIN
data/home/images/projects/bmn/ems-arch.png
Normal file
After Width: | Height: | Size: 363 KiB |
BIN
data/home/images/projects/bmn/ems-web.png
Normal file
After Width: | Height: | Size: 99 KiB |
BIN
data/home/images/projects/bmn/mon-arch.png
Normal file
After Width: | Height: | Size: 33 KiB |
BIN
data/home/images/projects/bmn/mon-db.png
Normal file
After Width: | Height: | Size: 204 KiB |
BIN
data/home/images/projects/bmn/nica.png
Normal file
After Width: | Height: | Size: 442 KiB |
BIN
data/home/images/projects/bmn/scs.png
Normal file
After Width: | Height: | Size: 166 KiB |
BIN
data/home/images/projects/bmn/sdp.png
Normal file
After Width: | Height: | Size: 41 KiB |
BIN
data/home/images/projects/bmn/vis-1.png
Normal file
After Width: | Height: | Size: 77 KiB |
BIN
data/home/images/projects/bmn/vis-2.png
Normal file
After Width: | Height: | Size: 182 KiB |