+ 
+
-
-
-
-
- {date} {title}
--
-
+
- + States: each device has a number of states that could be accessed by `getStatus` method. States could be either stored as some internal variables or calculated on demand. States calculation is synchronous! + +
- + Listeners: some external class which listens device state changes and events. By default listeners are represented by weak references so they could be finalized any time if not used. +
- + Connections: any external device connectors which are used by device. The difference between listener and connection is that device is obligated to notify all registered listeners about all changes, but connection is used by device at its own discretion. Also usually only one connection is used for each single purpose. + +
+ All strings are encoded in a following way: 2 bytes for string length, byte array of given length representing UTF-8 encoded string. +
+ + 1. Meta name as string. + 2. 2 bytes unsigned integer representing the number of values in the node. Could be zero. + 3. For each of values an encoded name of the value and then: + * `0` for Null value. + * `T` for `TIME` value, followed by two unsigned long values representing epoch second and nanosecond adjustment. + * `S` for `STRING` value followed by encoded string. + * `D` for double based `NUMBER` followed by double (double encoding specified by [Java serialization](https://docs.oracle.com/javase/8/docs/api/java/io/DataOutput.html#writeDouble-double-)). + * `I` for integer based `NUMBER` followed by signed 4-bytes integer value. + * `B` for [`BigDecimal`](https://docs.oracle.com/javase/8/docs/api/java/math/BigDecimal.html) based `NUMBER` followed by 2-byte unsigned integer defining the number of bytes in representation, bytes themselves and 4-byte scale factor. + * `+` for boolean `true`. + * `-` for boolean `false`. + * `L` for list value and then 2-byte list size followed by sequence of values (as in item 3, but without a name). + 4. 2 bytes unsigned integer representing the number of child nodes in this node. Could be zero. + 5. For each of child nodes: + * child node name, + * 2-byte unsigned integer representing number of children with this name. + * The sequence of nodes with given name encoding like the root node, but without name. \ No newline at end of file diff --git a/src/pages/content/dataforge/docs/envelopes/default_format.md b/src/pages/content/dataforge/docs/envelopes/default_format.md new file mode 100644 index 0000000..cfdc921 --- /dev/null +++ b/src/pages/content/dataforge/docs/envelopes/default_format.md @@ -0,0 +1,37 @@ +--- +content_type: "doc_shard" +title: "Streaming envelope format" +chapter: envelope +ordering: 1 +label: "envelope_format" +published: true +--- + +The default envelope format is developed for storage of binary data or transferring data via byte stream. The structure of this format is the following: + +* **Tag**. First 20 bytes of file or stream is reserved for envelope properties binary representation: + 1. `#~` - two ASCII symbols, beginning of binary string. + 2. 4 bytes - properties `type` field: envelope format type and version. For default format the string `DF02` is used, but in principle other envelope types could use the same format. + 3. 2 bytes - properties `metaType` field: metadata encoding type. + 4. 4 bytes - properties `metaLength` field: metadata length in bytes including new lines and other separators. + 5. 4 bytes - properties `dataLength` field: the data length in bytes. + 6. `~#` - two ASCII symbols, end of binary string. + 7. `\r\n` - two bytes, new line. + + The values are read as binary and transformed into 4-byte unsigned tag codes (Big endian). + +* **Metadata block**. Metadata in any accepted format. Additional formats could be provided by modules. The default metadata format is *UTF-8* encoded *XML* (tag code 0x584d). *JSON* format is provided by storage module. + + One must note that `metaLength` property is very important and in most cases is mandatory. It could be set to `0xffffffff` or `-1` value in order to force envelope reader to derive meta length automatically, but different readers do it in a different ways, so it strongly not recommended to do it if data block is not empty. + +* **Data block**. Any other data. If `dataLength` property is set to `0xffffffff` or `-1`, then it is supposed that data block ends with the end of file or stream. It is discouraged to use infinite data length for streaming data. Data block does not have any limitations for its content. It could even contain envelopes inside it! + +### Meta encoding + +DataForge server supports following metadata encoding types: + +1. **XML** encoding. The full name for this encoding is `XML`, the tag code is `XM`. +2. **JSON** encoding (currently supported only with `storage` module attached). The full name is `JSON`, the tag code is `JS`. +3. **Binary** encoding. DataForge own binary meta representation. The full name is `binary`, the tag code is `BI`. + +To avoid confusion. All full names are case insensitive. All meta is supposed to always use **UTF-8** character encoding. \ No newline at end of file diff --git a/src/pages/content/dataforge/docs/envelopes/envelope_description.md b/src/pages/content/dataforge/docs/envelopes/envelope_description.md new file mode 100644 index 0000000..e237e1e --- /dev/null +++ b/src/pages/content/dataforge/docs/envelopes/envelope_description.md @@ -0,0 +1,18 @@ +--- +content_type: "doc_shard" +title: "Envelope description" +chapter: envelope +ordering: 20 +label: "envelope_description" +published: true +--- +Interpretation of envelope contents is basically left for user, but for convenience purposes there is convention for envelope description encoded inside meta block. + +The envelope description is placed into hidden `@envelope` meta node. The description could contain following values: + +- `@envelope.type` (STRING): Type of the envelope content +- `@envelope.dataType` (STRING): Type of the envelope data +- `@envelope.description` (STRING): Description of the envelope content +- `@envelope.time` (TIME): Time of envelope creation + +Both envelope type and data type are supposed to be presented in reversed Internet domain name like java packages. diff --git a/src/pages/content/dataforge/docs/envelopes/envelopes.md b/src/pages/content/dataforge/docs/envelopes/envelopes.md new file mode 100644 index 0000000..7cd91ce --- /dev/null +++ b/src/pages/content/dataforge/docs/envelopes/envelopes.md @@ -0,0 +1,19 @@ +--- +content_type: "doc_shard" +title: "Envelopes" +chapter: envelope +label: "envelopes" +version: 1.0 +date: 2016-02-08 +published: true +--- + +The DataForge functionality is largely based on metadata exchange and therefore the main medium for messages between different parts of the system is `Meta` object and its derivatives. But sometimes one needs not only to transfer metadata but some binary or object data as well. For this DataForge supports an 'Envelope' entity, which contains both meta block and binary data block. Envelope could be automatically serialized to or from a byte stream. + +DataForge supports an extensible list of Envelope encoding methods. Specific method is defined by so-called encoding *properties* - the list of key-value pairs that define the specific way the meta and data are encoded. Meta-data itself also could have different encoding. Out of the box DataForge server supports two envelope formats and three meta formats. + + + + + + \ No newline at end of file diff --git a/src/pages/content/dataforge/docs/envelopes/tagless_format.md b/src/pages/content/dataforge/docs/envelopes/tagless_format.md new file mode 100644 index 0000000..0a25471 --- /dev/null +++ b/src/pages/content/dataforge/docs/envelopes/tagless_format.md @@ -0,0 +1,25 @@ +--- +content_type: "doc_shard" +title: "Tagless envelope format" +chapter: envelope +ordering: 2 +label: "tagless_format" +version: 1.0 +published: true +--- +Tagless format is developed to store textual data without need for binary block at the beginning. It is not recommended to use it for binary data or for streaming. The structure of this format is the following: + + +1. **The header line**. `#~DFTL~#`. The line is used only for identify the DataForge envelope. Standard reader is also configured to skip any lines starting with # before this line, so it is compatible with [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)). All header lines in tagless format must have at least new line `\n` character after them (DOS/Windows new line `\r\n` is also supported). + +2. **Properties**. Properties are defined in a textual form. Each property is defined in its own lined in a following way: + + ```#?
+**Note** that putting elements or values to configuration follows the same naming convention as getting from it. +Meaning putting to `some_name.something` will actually create or put to the node `some_name` if it exists. +Otherwise, new node is created. +
diff --git a/src/pages/content/dataforge/docs/meta/hidden_values.md b/src/pages/content/dataforge/docs/meta/hidden_values.md new file mode 100644 index 0000000..9ed2f6a --- /dev/null +++ b/src/pages/content/dataforge/docs/meta/hidden_values.md @@ -0,0 +1,13 @@ +--- +content_type: "doc_shard" +title: "Hidden values" +chapter: "meta" +ordering: 10 +label: "hidden_values" +version: 1.0 +published: true +--- + +Meta structure supports so-called *hidden* values and nodes. These values and nodes exist in the meta and could be requested either by user or framework, but are not shown by casual listing. These values and nodes are considered *system* and are not intended to be defined by user in most cases. + +Hidden values and nodes names starts with symbol `@` like `@node`. Not all meta-data representations allow this symbol so in could be replaced by escaped sequence `_at_` in text format (`<_at_node/>` in XML). \ No newline at end of file diff --git a/src/pages/content/dataforge/docs/meta/meta-data.md b/src/pages/content/dataforge/docs/meta/meta-data.md new file mode 100644 index 0000000..8d6455c --- /dev/null +++ b/src/pages/content/dataforge/docs/meta/meta-data.md @@ -0,0 +1,33 @@ +--- +content_type: "doc_shard" +title: "Meta-data processor" +chapter: "meta" +label: "meta" +version: 2.0 +date: 2015-08-27 +published: true +--- +The main and the most distinguishing feature of DataForge framework is a concept of data analysis as a meta-data processor. +First of all, one need to define main terms: + +* **Data**. Any information from experiment or simulation provided by user. The data format is not governed by DataForge so it could be presented in any form. The important point concerning data is that data is by default immutable. Meaning the program could create its modified copies (which is not recommended), but could not modify initial input in any way. + +* **Meta-data**. Meta-data contrary to data has a fixed internal representation provided by Meta object. Meta is simple [tree-like object](#meta_structure) that can conveniently store values and Meta nodes. Meta-data is either provided by user or generated in analysis process. Inside the analysis process Meta is immutable, but for some purposes, changeable meta nodes could be used during analysis configuration and results representation (see [Configuration](#configuration)). + +The fundamental point of the whole DataForge philosophy is that every external input could be either data or meta-data. This point could seem dull at first glance, but in fact, it have a number of very important consequences: + +* *No scripts*. DataForge encourages user not to use imperative code to manipulate data. Any procedure to be performed on data should be presented either by some hard-coded meta-data processor rule (the function that takes data and metadata and produces some output without any additional external information) or as declarative process definition in form of meta-data. Since the most of data analysis nowadays is made by different scripts, the loss of scripting capability could seem to be a tremendous blow to framework functionality, but in fact every thing one could do with script, one also can do with declaration and declaration processing rules. Also purely declarative description allows for easy error check and analysis scaling (via parallelism). + +
+ + **Note:** Those, who still like scripting, can still either use DataForge as a library, or use [GRIND](#grind) + +
+ +* *Automatic scaling*. Since particular analysis of some piece of data depends only on that piece and on it's meta-data, such analysis could be easily scaled for any number of such data pieces. + +* *Meta composition*. Since the configuration of analysis and meta-data ara both have fixed representation, one could easily combine specific analysis configuration from different parts taken from different sources. For example, one could have a general configuration for the whole data set and specific changes for some specific point. One do not have to write a specific script to work with this particular point, just override its meta-data! + +* *No global state*. Since neither data, nor meta-data account for global states, there are no global states! The closest to the global states as you get is context variables (see [context](#context)), which are used to connect analysis process to system environment, but these states are not used in the analysis itself. + +* *Automatic data flow organisation*. Since user does not in general control the order and time of actions performed by the framework, the work of arranging data flow, result caching, etc. could be actually done by the framework itself. \ No newline at end of file diff --git a/src/pages/content/dataforge/docs/meta/meta-layers.md b/src/pages/content/dataforge/docs/meta/meta-layers.md new file mode 100644 index 0000000..ad2855a --- /dev/null +++ b/src/pages/content/dataforge/docs/meta/meta-layers.md @@ -0,0 +1,22 @@ +--- +content_type: "doc_shard" +title: "Meta-data composition" +chapter: "meta" +ordering: 5 +label: "meta-layers" +version: 1.0 +published: true +--- + +An important part of working with meta is *composition*. Let us work with two use-cases: + +1. There is a data supplied with meta and one needs modified version of the meta. +2. There is a list of data with the same meta and one needs to change meta only for one data piece. + +DataForge provides instruments to easily modify meta (or, since meta is immutable, create a modified instance), but it is not a good solution. A much better way is to use instrument called `Laminate`. Laminate implements meta specification, but stores not one tree of values, but multiple layers of meta. When one requests a Laminate for a value ore meta, it automatically forwards request to the first meta layer. If required node or value is not found in the first layer, the request is forwarded to second layer etc. Laminate also contains meta descriptor, which could be used for default values. Laminate functionality also allows to use information from all of its layers, for example join lists of nodes instead of replacing them. Laminate layers also could be merged together to create classical meta and increase performance. Of course, in this case special features like custom use of all layers is lost. + +Using Laminate in case 1 looks like this: if one needs just to change or add some value, one creates a Laminate with initial meta as second layer and override layer containing only values to be changed as first. + +For case 2 the solution is even simpler: `DataNode` structures automatically uses laminates to define meta for specific data pieces. So one needs just to define meta for specific data, it will be automatically layered with node meta (or multiple meta elements if node data node structure has many levels). + +The typical usage of data layering could be demonstrated on `Actions` (push data flow). Action has 3 arguments: `Context`, `DataNode` and `Meta`. The actual configuration for specific `Data` is defined as a laminate containing data meta layer, node meta layer(s) and action meta layer. Meaning that actual action meta is used only if appropriate positions are not defined in data meta (data knows best about how it should be analyzed). diff --git a/src/pages/content/dataforge/docs/meta/meta_structure.md b/src/pages/content/dataforge/docs/meta/meta_structure.md new file mode 100644 index 0000000..4d9d94a --- /dev/null +++ b/src/pages/content/dataforge/docs/meta/meta_structure.md @@ -0,0 +1,42 @@ +--- +content_type: "doc_shard" +title: "Meta-data object structure" +chapter: "meta" +ordering: 4 +label: "meta_structure" +version: 1.1 +date: 2015-08-27 +published: true +--- +
+ 
+ |
+
+ + The Meta object is a tree-like structure, which can contain other meta objects as branches (which are + called + elements) and Value objects as leafs. + Both Values and Meta elements are organized in String-keyed maps. And each map element is a list of + appropriate type. By requesting single Value or Meta element one is supposed to request first element of + this list. + ++ Note that such lists are always immutable. Trying to change it may cause a error. ++ + While meta itself does not have any write methods and is considered to be immutable, some of its + extensions do have methods that can change meta structure. One should be careful not to use mutable meta + elements when one need immutable one. + ++ In order to conveniently edit meta, there is MetaBuilder class. + + |
+
+
+
-
+
)
export default AppPage
\ No newline at end of file
diff --git a/src/pages/dataforge/misc.js b/src/pages/dataforge/misc.js
index e69de29..715b3d8 100644
--- a/src/pages/dataforge/misc.js
+++ b/src/pages/dataforge/misc.js
@@ -0,0 +1,76 @@
+import React from "react"
+import Layout from "../../components/dataforge/layout"
+
+import sms_logo from "../../images/dataforge/sms_logo.png"
+
+const MiscPage = () => (
+ Troitsk nu-mass
-
+
The main aim of Troitsk nu-mass is a search for masses of active and sterile neutrinos in the beta-spectrum of tritium.
-
About us...
+ ++ +
Other data analysis framework
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ +
 | */}
+
+ The java analysis studio is quite advanced data visualization framework developed in Standford accelerator center. It is modular and has good object oriented design +It is intended to make an interface to integrate DataForge framework with JAS3 visualization via AIDA plugin + |
+
+
|
+
+ |
+
+ + DataMelt is huge collection of scientific libraries accompanied by visual framework which allows to + write analysis scripts in different JVM languages. It is a good memory-safe alternative to Root, but + sadly it inherits a lot of Root architectural drawbacks as well. + + |
+
+
-
-
- )
+ const title = post.frontmatter.title;
+ const date = post.frontmatter.date;
+ return(
+ -
+
+ )
})}
+
+ + {title}{date} +
+ +Versioning
+DataForge uses Maven version policy.
+Source code and documentation consistency
+Be warned that for SNAPSHOT versions source code and documentation could be slightly inconsistent due to not synchronized update of this site.
+ ++ +
| Version | +Javadoc | +Release notes | +Info | +
|---|---|---|---|
| 0.1.0 | ++ | + | + |
| 0.1.1 | ++ | + |
+
|
+
| 0.4.0 - SNAPSHOT | ++ HTML + | ++ | For list of features see documentation |
+