Merge pull request 'v0.10.0' (!81) from dev into master

Reviewed-on: #81

.gitignore

@@ -5,5 +5,7 @@ out/
 .gradle
 build/
 
+.kotlin
+
 
 !gradle-wrapper.jar

CHANGELOG.md

@@ -14,6 +14,94 @@
 
 ### Security
 
+## 0.10.0 - 2025-01-19
+
+### Added
+
+- Coroutine exception logging in context
+- `ObservableMutableMetaSerializer`
+- `MutableMetaView` - a Meta wrapper that creates nodes only when its or its children are changed. 
+
+### Changed
+
+- Simplify inheritance logic in `MutableTypedMeta`
+- Full rework of `DataTree` and associated interfaces (`DataSource`, `DataSink`, etc.).
+- Filter data by type is moved from `dataforge-data` to `dataforge-workspace` to avoid reflection dependency.
+
+### Deprecated
+
+- MetaProvider `spec` is replaced by `readable`. `listOfSpec` replaced with `listOfReadable`
+
+### Removed
+
+- Remove implicit io format resolver in `IOPlugin` and `FileWorkspaceCache`. There are no guarantees that only one format is present in the contrxt for each type.
+- Dependencies on `atomicfu` and `kotlin.reflect` from dataforge-data to improve performance.
+
+### Fixed
+
+- Fixed NameToken parsing.
+- Top level string list meta conversion.
+
+## 0.9.0 - 2024-06-04
+
+### Added
+
+- Custom CoroutineContext during `Context` creation.
+
+### Changed
+
+- Kotlin 2.0
+- `MetaSpec` renamed to `MetaReader`. MetaSpec is now reserved for builder-based generation of meta descriptors.
+- Add self-type for Meta. Remove unsafe cast method for meta instances.
+
+### Removed
+
+- Automatic descriptors for schema. It is not possible to implement them without heavy reflection.
+
+## 0.8.2 - 2024-04-27
+
+### Added
+
+- Name index comparator
+- Specialized ByteArrayValue
+
+### Changed
+
+- DataSink `branch` is replaced with `putAll` to avoid confusion with DataTree methods
+- Meta delegate now uses a specific class that has a descriptor
+
+### Fixed
+
+- `listOfScheme` and `listOfConvertable` delegates provides correct items order.
+- Scheme meta setter works with proper sub-branch.
+- NameToken.parse improper work with indices.
+- Proper data handling for cache.
+
+## 0.8.0 - 2024-02-03
+
+### Added
+
+- Wasm artifacts
+- Add automatic MetaConverter for serializeable objects
+- Add Meta and MutableMeta delegates for convertable and serializeable
+- Meta mapping for data.
+
+### Changed
+
+- Descriptor `children` renamed to `nodes`
+- `MetaConverter` now inherits `MetaSpec` (former `Specifiction`). So `MetaConverter` could be used more universally.
+- Meta copy and modification now use lightweight non-observable meta builders.
+- Full refactor of Data API. DataTree now works similar to Meta: contains optional anonymous root element and data items. Updates are available for `ObservaleDataSource` and `ObservableDataTree` variants.
+
+### Deprecated
+
+- `node(key,converter)` in favor of `serializable` delegate
+
+### Fixed
+
+- Partially fixed a bug with `MutableMeta` observable wrappers.
+- `valueSequence` now include root value. So `meta.update` works properly.
+
 ## 0.7.0 - 2023-11-26
 
 ### Added

README.md

@@ -1,7 +1,70 @@
 [![JetBrains Research](https://jb.gg/badges/research.svg)](https://confluence.jetbrains.com/display/ALL/JetBrains+on+GitHub)
 [![DOI](https://zenodo.org/badge/148831678.svg)](https://zenodo.org/badge/latestdoi/148831678)
 
-![Gradle build](https://github.com/mipt-npm/dataforge-core/workflows/Gradle%20build/badge.svg)
+## Publications
+
+* [A general overview](https://doi.org/10.1051/epjconf/201817705003)
+* [An application in "Troitsk nu-mass" experiment](https://doi.org/10.1088/1742-6596/1525/1/012024)
+
+## Video
+
+* [A presentation on application of DataForge (legacy version) to Troitsk nu-mass analysis.](https://youtu.be/OpWzLXUZnLI?si=3qn7EMruOHMJX3Bc)
+
+## Questions and Answers
+
+In this section, we will try to cover DataForge main ideas in the form of questions and answers.
+
+### General
+
+**Q**: I have a lot of data to analyze. The analysis process is complicated, requires a lot of stages, and data flow is not always obvious. Also, the data size is huge, so I don't want to perform operation I don't need (calculate something I won't need or calculate something twice). I need it to be performed in parallel and probably on remote computer. By the way, I am sick and tired of scripts that modify other scripts that control scripts. Could you help me?
+
+**A**: Yes, that is precisely the problem DataForge was made to solve. It allows performing some automated data manipulations with optimization and parallelization. The important thing that data processing recipes are made in the declarative way, so it is quite easy to perform computations on a remote station. Also, DataForge guarantees reproducibility of analysis results.
+
+**Q**: How does it work?
+
+**A**: At the core of DataForge lies the idea of metadata processor. It utilizes the fact that to analyze something you need data itself and some additional information about what does that data represent and what does user want as a result. This additional information is called metadata and could be organized in a regular structure (a tree of values similar to XML or JSON). The important thing is that this distinction leaves no place for user instructions (or scripts). Indeed, the idea of DataForge logic is that one does not need imperative commands. The framework configures itself according to input meta-data and decides what operations should be performed in the most efficient way.
+
+**Q**: But where does it take algorithms to use?
+
+**A**: Of course algorithms must be written somewhere. No magic here. The logic is written in specialized modules. Some modules are provided out of the box at the system core, some need to be developed for a specific problem.
+
+**Q**: So I still need to write the code? What is the difference then?
+
+**A**: Yes, someone still needs to write the code. But not necessary you. Simple operations could be performed using provided core logic. Also, your group can have one programmer writing the logic and all other using it without any real programming expertise. The framework organized in a such way that one writes some additional logic, they do not need to think about complicated thing like parallel computing, resource handling, logging, caching, etc. Most of the things are done by the DataForge.
+
+### Platform
+
+**Q**: Which platform does DataForge use? Which operating system is it working on?
+
+**A**: The DataForge is mostly written in Kotlin-multiplatform and could be used on JVM, JS and native targets. Some modules and functions are supported only on JVM
+
+**Q**: Can I use my C++/Fortran/Python code in DataForge?
+
+**A**: Yes, as long as the code could be called from Java. Most common languages have a bridge for Java access. There are completely no problems with compiled C/Fortran libraries. Python code could be called via one of existing python-java interfaces. It is also planned to implement remote method invocation for common languages, so your Python, or, say, Julia, code could run in its native environment. The metadata processor paradigm makes it much easier to do so.
+
+### Features
+
+**Q**: What other features does DataForge provide?
+
+**A**: Alongside metadata processing (and a lot of tools for metadata manipulation and layering), DataForge has two additional important concepts:
+
+* **Modularisation**. Contrary to lot other frameworks, DataForge is intrinsically modular. The mandatory part is a rather tiny core module. Everything else could be customized.
+
+* **Context encapsulation**. Every DataForge task is executed in some context. The context isolates environment for the task and also works as dependency injection base and specifies interaction of the task with the external world.
+
+### Misc
+
+**Q**: So everything looks great, can I replace my ROOT / other data analysis framework with DataForge?
+
+**A**: One must note that DataForge is made for analysis, not for visualization. The visualization and user interaction capabilities of DataForge are rather limited compared to frameworks like ROOT, JAS3 or DataMelt. The idea is to provide reliable API and core functionality. [VisionForge](https://git.sciprog.center/kscience/visionforge) project aims to provide tools for both 2D and 3D visualization both locally and remotely.
+
+**Q**: How does DataForge compare to cluster computation frameworks like Apache Spark?
+
+**A**: It is not the purpose of DataForge to replace cluster computing software. DataForge has some internal parallelism mechanics and implementations, but they are most certainly worse than specially developed programs. Still, DataForge is not fixed on one single implementation. Your favourite parallel processing tool could be still used as a back-end for the DataForge. With full benefit of configuration tools, integrations and no performance overhead.
+
+**Q**: Is it possible to use DataForge in notebook mode?
+
+**A**: [Kotlin jupyter](https://github.com/Kotlin/kotlin-jupyter) allows using any JVM program in a notebook mode. The dedicated module for DataForge is work in progress.
 
 
 ### [dataforge-context](dataforge-context)
@@ -14,16 +77,31 @@
 > **Maturity**: EXPERIMENTAL
 
 ### [dataforge-io](dataforge-io)
-> IO module
+> Serialization foundation for Meta objects and Envelope processing.
 >
 > **Maturity**: EXPERIMENTAL
+>
+> **Features:**
+> - [IO format](dataforge-io/src/commonMain/kotlin/space/kscience/dataforge/io/IOFormat.kt) : A generic API for reading something from binary representation and writing it to Binary.
+> - [Binary](dataforge-io/src/commonMain/kotlin/space/kscience/dataforge/io/Binary.kt) : Multi-read random access binary.
+> - [Envelope](dataforge-io/src/commonMain/kotlin/space/kscience/dataforge/io/Envelope.kt) : API and implementations for combined data and metadata format.
+> - [Tagged envelope](dataforge-io/src/commonMain/kotlin/space/kscience/dataforge/io/TaggedEnvelope.kt) : Implementation for binary-friendly envelope format with machine readable tag and forward size declaration.
+> - [Tagged envelope](dataforge-io/src/commonMain/kotlin/space/kscience/dataforge/io/TaglessEnvelope.kt) : Implementation for text-friendly envelope format with text separators for sections.
+
 
 ### [dataforge-meta](dataforge-meta)
-> Meta definition and basic operations on meta
+> Core Meta and Name manipulation module
 >
 > **Maturity**: DEVELOPMENT
+>
+> **Features:**
+> - [Meta](dataforge-meta/src/commonMain/kotlin/space/kscience/dataforge/meta/Meta.kt) : **Meta** is the representation of basic DataForge concept: Metadata, but it also could be called meta-value tree.
+> - [Value](dataforge-meta/src/commonMain/kotlin/space/kscience/dataforge/meta/Value.kt) : **Value** a sum type for different meta values.
+> - [Name](dataforge-meta/src/commonMain/kotlin/space/kscience/dataforge/names/Name.kt) : **Name** is an identifier to access tree-like structure.
+
 
 ### [dataforge-scripting](dataforge-scripting)
+> Scripting definition fow workspace generation
 >
 > **Maturity**: PROTOTYPE
 
@@ -31,6 +109,11 @@
 >
 > **Maturity**: EXPERIMENTAL
 
+### [dataforge-io/dataforge-io-proto](dataforge-io/dataforge-io-proto)
+> ProtoBuf Meta representation
+>
+> **Maturity**: PROTOTYPE
+
 ### [dataforge-io/dataforge-io-yaml](dataforge-io/dataforge-io-yaml)
 > YAML meta converters and Front Matter envelope format
 >

build.gradle.kts

@@ -3,24 +3,31 @@ import space.kscience.gradle.useApache2Licence
 import space.kscience.gradle.useSPCTeam
 
 plugins {
-    id("space.kscience.gradle.project")
+    alias(spclibs.plugins.kscience.project)
+    alias(spclibs.plugins.kotlinx.kover)
 }
 
 allprojects {
     group = "space.kscience"
-    version = "0.7.0"
+    version = "0.10.0"
 }
 
 subprojects {
     apply(plugin = "maven-publish")
 
     tasks.withType<KotlinCompile> {
-        kotlinOptions {
+        compilerOptions {
-            freeCompilerArgs = freeCompilerArgs + "-Xcontext-receivers"
+            freeCompilerArgs.add("-Xcontext-receivers")
         }
     }
 }
 
+dependencies{
+    subprojects.forEach {
+        dokka(it)
+    }
+}
+
 readme {
     readmeTemplate = file("docs/templates/README-TEMPLATE.md")
 }
@@ -30,8 +37,8 @@ ksciencePublish {
         useApache2Licence()
         useSPCTeam()
     }
-    repository("spc","https://maven.sciprog.center/kscience")
+    repository("spc", "https://maven.sciprog.center/kscience")
-    sonatype()
+    central()
 }
 
 apiValidation {

dataforge-context/README.md

@@ -6,18 +6,16 @@ Context and provider definitions
 
 ## Artifact:
 
-The Maven coordinates of this project are `space.kscience:dataforge-context:0.7.0`.
+The Maven coordinates of this project are `space.kscience:dataforge-context:0.10.0`.
 
 **Gradle Kotlin DSL:**
 ```kotlin
 repositories {
     maven("https://repo.kotlin.link")
-    //uncomment to access development builds
-    //maven("https://maven.pkg.jetbrains.space/spc/p/sci/dev")
     mavenCentral()
 }
 
 dependencies {
-    implementation("space.kscience:dataforge-context:0.7.0")
+    implementation("space.kscience:dataforge-context:0.10.0")
 }
 ```

dataforge-context/api/dataforge-context.api

@@ -57,6 +57,7 @@ public abstract interface class space/kscience/dataforge/context/ContextAware {
 
 public final class space/kscience/dataforge/context/ContextBuilder {
 	public final fun build ()Lspace/kscience/dataforge/context/Context;
+	public final fun coroutineContext (Lkotlin/coroutines/CoroutineContext;)V
 	public final fun getName ()Lspace/kscience/dataforge/names/Name;
 	public final fun plugin (Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Lkotlin/jvm/functions/Function1;)V
 	public final fun plugin (Lspace/kscience/dataforge/context/Plugin;)V
@@ -69,9 +70,6 @@ public final class space/kscience/dataforge/context/ContextBuilder {
 	public final fun properties (Lkotlin/jvm/functions/Function1;)V
 }
 
-public final class space/kscience/dataforge/context/ContextBuilderKt {
-}
-
 public final class space/kscience/dataforge/context/DefaultLogManager : space/kscience/dataforge/context/AbstractPlugin, space/kscience/dataforge/context/LogManager {
 	public static final field Companion Lspace/kscience/dataforge/context/DefaultLogManager$Companion;
 	public fun <init> ()V
@@ -212,14 +210,14 @@ public final class space/kscience/dataforge/context/PluginTag : space/kscience/d
 	public fun toString ()Ljava/lang/String;
 }
 
-public final class space/kscience/dataforge/context/PluginTag$$serializer : kotlinx/serialization/internal/GeneratedSerializer {
+public synthetic class space/kscience/dataforge/context/PluginTag$$serializer : kotlinx/serialization/internal/GeneratedSerializer {
 	public static final field INSTANCE Lspace/kscience/dataforge/context/PluginTag$$serializer;
-	public fun childSerializers ()[Lkotlinx/serialization/KSerializer;
+	public final fun childSerializers ()[Lkotlinx/serialization/KSerializer;
 	public synthetic fun deserialize (Lkotlinx/serialization/encoding/Decoder;)Ljava/lang/Object;
-	public fun deserialize (Lkotlinx/serialization/encoding/Decoder;)Lspace/kscience/dataforge/context/PluginTag;
+	public final fun deserialize (Lkotlinx/serialization/encoding/Decoder;)Lspace/kscience/dataforge/context/PluginTag;
-	public fun getDescriptor ()Lkotlinx/serialization/descriptors/SerialDescriptor;
+	public final fun getDescriptor ()Lkotlinx/serialization/descriptors/SerialDescriptor;
 	public synthetic fun serialize (Lkotlinx/serialization/encoding/Encoder;Ljava/lang/Object;)V
-	public fun serialize (Lkotlinx/serialization/encoding/Encoder;Lspace/kscience/dataforge/context/PluginTag;)V
+	public final fun serialize (Lkotlinx/serialization/encoding/Encoder;Lspace/kscience/dataforge/context/PluginTag;)V
 	public fun typeParametersSerializers ()[Lkotlinx/serialization/KSerializer;
 }
 
@@ -249,13 +247,19 @@ public final class space/kscience/dataforge/context/SlfLogManager$Companion : sp
 	public fun getTag ()Lspace/kscience/dataforge/context/PluginTag;
 }
 
-public final class space/kscience/dataforge/properties/PropertyKt {
+public abstract interface annotation class space/kscience/dataforge/descriptors/Description : java/lang/annotation/Annotation {
+	public abstract fun value ()Ljava/lang/String;
 }
 
-public final class space/kscience/dataforge/properties/SchemePropertyKt {
+public abstract interface annotation class space/kscience/dataforge/descriptors/DescriptorResource : java/lang/annotation/Annotation {
+	public abstract fun resourceName ()Ljava/lang/String;
 }
 
-public final class space/kscience/dataforge/provider/DfTypeKt {
+public abstract interface annotation class space/kscience/dataforge/descriptors/DescriptorUrl : java/lang/annotation/Annotation {
+	public abstract fun url ()Ljava/lang/String;
+}
+
+public abstract interface annotation class space/kscience/dataforge/descriptors/Multiple : java/lang/annotation/Annotation {
 }
 
 public final class space/kscience/dataforge/provider/Path : java/lang/Iterable, kotlin/jvm/internal/markers/KMappedMarker {
@@ -278,6 +282,7 @@ public final class space/kscience/dataforge/provider/Path : java/lang/Iterable,
 
 public final class space/kscience/dataforge/provider/Path$Companion {
 	public final fun parse-X5wN5Vs (Ljava/lang/String;)Ljava/util/List;
+	public final fun serializer ()Lkotlinx/serialization/KSerializer;
 }
 
 public final class space/kscience/dataforge/provider/PathKt {

dataforge-context/build.gradle.kts

@@ -8,14 +8,15 @@ kscience {
     jvm()
     js()
     native()
+    wasm()
     useCoroutines()
     useSerialization()
-    dependencies {
+    commonMain {
-        api(project(":dataforge-meta"))
+        api(projects.dataforgeMeta)
     }
-    dependencies(jvmMain){
+    jvmMain{
-        api(kotlin("reflect"))
+        api(spclibs.kotlin.reflect)
-        api("org.slf4j:slf4j-api:1.7.30")
+        api(spclibs.slf4j)
     }
 }
 

dataforge-context/src/commonMain/kotlin/space/kscience/dataforge/context/Context.kt

@@ -1,5 +1,6 @@
 package space.kscience.dataforge.context
 
+import kotlinx.coroutines.CoroutineExceptionHandler
 import kotlinx.coroutines.CoroutineScope
 import kotlinx.coroutines.Job
 import kotlinx.coroutines.SupervisorJob
@@ -10,6 +11,7 @@ import space.kscience.dataforge.misc.ThreadSafe
 import space.kscience.dataforge.names.Name
 import space.kscience.dataforge.provider.Provider
 import kotlin.coroutines.CoroutineContext
+import kotlin.coroutines.EmptyCoroutineContext
 
 /**
  * The local environment for anything being done in DataForge framework. Contexts are organized into tree structure with [Global] at the top.
@@ -26,6 +28,7 @@ public open class Context internal constructor(
     public val parent: Context?,
     plugins: Set<Plugin>, // set of unattached plugins
     meta: Meta,
+    coroutineContext: CoroutineContext = EmptyCoroutineContext,
 ) : Named, MetaRepr, Provider, CoroutineScope {
 
     /**
@@ -65,7 +68,9 @@ public open class Context internal constructor(
 
     override val coroutineContext: CoroutineContext by lazy {
         (parent ?: Global).coroutineContext.let { parenContext ->
-            parenContext + SupervisorJob(parenContext[Job])
+            parenContext + coroutineContext + SupervisorJob(parenContext[Job]) + CoroutineExceptionHandler { _, throwable ->
+                logger.error(throwable) { "Exception in context $name" }
+            }
         }
     }
 

dataforge-context/src/commonMain/kotlin/space/kscience/dataforge/context/ContextBuilder.kt

@@ -13,6 +13,8 @@ import space.kscience.dataforge.names.plus
 import kotlin.collections.component1
 import kotlin.collections.component2
 import kotlin.collections.set
+import kotlin.coroutines.CoroutineContext
+import kotlin.coroutines.EmptyCoroutineContext
 
 /**
  * A convenience builder for context
@@ -59,8 +61,15 @@ public class ContextBuilder internal constructor(
         plugin(DeFactoPluginFactory(plugin))
     }
 
+    private var coroutineContext: CoroutineContext = EmptyCoroutineContext
+
+    public fun coroutineContext(coroutineContext: CoroutineContext) {
+        this.coroutineContext = coroutineContext
+    }
+
+
     public fun build(): Context {
-        val contextName = name ?: NameToken("@auto",hashCode().toUInt().toString(16)).asName()
+        val contextName = name ?: NameToken("@auto", hashCode().toUInt().toString(16)).asName()
         val plugins = HashMap<PluginTag, Plugin>()
 
         fun addPlugin(factory: PluginFactory<*>, meta: Meta) {
@@ -86,7 +95,7 @@ public class ContextBuilder internal constructor(
             addPlugin(factory, meta)
         }
 
-        return Context(contextName, parent, plugins.values.toSet(), meta.seal())
+        return Context(contextName, parent, plugins.values.toSet(), meta.seal(), coroutineContext)
     }
 }
 

dataforge-context/src/commonMain/kotlin/space/kscience/dataforge/context/Plugin.kt

@@ -3,7 +3,7 @@ package space.kscience.dataforge.context
 import space.kscience.dataforge.context.Plugin.Companion.TARGET
 import space.kscience.dataforge.meta.Meta
 import space.kscience.dataforge.meta.MetaRepr
-import space.kscience.dataforge.misc.DfId
+import space.kscience.dataforge.misc.DfType
 import space.kscience.dataforge.misc.Named
 import space.kscience.dataforge.names.Name
 import space.kscience.dataforge.names.parseAsName
@@ -18,7 +18,7 @@ import space.kscience.dataforge.provider.Provider
  *
  * create - configure - attach - detach - destroy
  */
-@DfId(TARGET)
+@DfType(TARGET)
 public interface Plugin : Named, ContextAware, Provider, MetaRepr {
 
     /**

dataforge-context/src/commonMain/kotlin/space/kscience/dataforge/context/PluginFactory.kt

@@ -1,9 +1,9 @@
 package space.kscience.dataforge.context
 
 import space.kscience.dataforge.meta.Meta
-import space.kscience.dataforge.misc.DfId
+import space.kscience.dataforge.misc.DfType
 
-@DfId(PluginFactory.TYPE)
+@DfType(PluginFactory.TYPE)
 public interface PluginFactory<T : Plugin> : Factory<T> {
     public val tag: PluginTag
 

dataforge-context/src/commonMain/kotlin/space/kscience/dataforge/properties/MetaProperty.kt

@@ -1,35 +0,0 @@
-package space.kscience.dataforge.properties
-
-
-import space.kscience.dataforge.meta.Meta
-import space.kscience.dataforge.meta.ObservableMutableMeta
-import space.kscience.dataforge.meta.transformations.MetaConverter
-import space.kscience.dataforge.meta.transformations.nullableMetaToObject
-import space.kscience.dataforge.meta.transformations.nullableObjectToMeta
-import space.kscience.dataforge.misc.DFExperimental
-import space.kscience.dataforge.names.Name
-import space.kscience.dataforge.names.startsWith
-
-@DFExperimental
-public class MetaProperty<T : Any>(
-    public val meta: ObservableMutableMeta,
-    public val name: Name,
-    public val converter: MetaConverter<T>,
-) : Property<T?> {
-
-    override var value: T?
-        get() = converter.nullableMetaToObject(meta[name])
-        set(value) {
-            meta[name] = converter.nullableObjectToMeta(value) ?: Meta.EMPTY
-        }
-
-    override fun onChange(owner: Any?, callback: (T?) -> Unit) {
-        meta.onChange(owner) { name ->
-            if (name.startsWith(this@MetaProperty.name)) callback(converter.nullableMetaToObject(this[name]))
-        }
-    }
-
-    override fun removeChangeListener(owner: Any?) {
-        meta.removeListener(owner)
-    }
-}

dataforge-context/src/commonMain/kotlin/space/kscience/dataforge/properties/Property.kt

@@ -1,47 +0,0 @@
-package space.kscience.dataforge.properties
-
-import kotlinx.coroutines.ExperimentalCoroutinesApi
-import kotlinx.coroutines.flow.MutableStateFlow
-import kotlinx.coroutines.flow.StateFlow
-import space.kscience.dataforge.misc.DFExperimental
-
-@DFExperimental
-public interface Property<T> {
-    public var value: T
-
-    public fun onChange(owner: Any? = null, callback: (T) -> Unit)
-    public fun removeChangeListener(owner: Any? = null)
-}
-
-@DFExperimental
-@OptIn(ExperimentalCoroutinesApi::class)
-public fun <T> Property<T>.toFlow(): StateFlow<T> = MutableStateFlow(value).also { stateFlow ->
-    onChange {
-        stateFlow.value = it
-    }
-}
-
-/**
- * Reflect all changes in the [source] property onto this property. Does not reflect changes back.
- *
- * @return a mirroring job
- */
-@DFExperimental
-public fun <T> Property<T>.mirror(source: Property<T>) {
-    source.onChange(this) {
-        this.value = it
-    }
-}
-
-/**
- * Bi-directional connection between properties
- */
-@DFExperimental
-public fun <T> Property<T>.bind(other: Property<T>) {
-    onChange(other) {
-        other.value = it
-    }
-    other.onChange {
-        this.value = it
-    }
-}

dataforge-context/src/commonMain/kotlin/space/kscience/dataforge/properties/metaAsFlow.kt

@@ -0,0 +1,51 @@
+package space.kscience.dataforge.properties
+
+
+import kotlinx.coroutines.CoroutineScope
+import kotlinx.coroutines.Job
+import kotlinx.coroutines.channels.awaitClose
+import kotlinx.coroutines.flow.*
+import kotlinx.coroutines.launch
+import space.kscience.dataforge.meta.*
+import space.kscience.dataforge.misc.DFExperimental
+
+@DFExperimental
+public fun <T> ObservableMeta.asFlow(converter: MetaReader<T>): Flow<T> = callbackFlow {
+    onChange(this){
+        trySend(converter.read(this))
+    }
+
+    awaitClose{
+        removeListener(this)
+    }
+}
+
+@DFExperimental
+public fun <T> MutableMeta.listenTo(
+    scope: CoroutineScope,
+    converter: MetaConverter<T>,
+    flow: Flow<T>,
+): Job = flow.onEach {
+    update(converter.convert(it))
+}.launchIn(scope)
+
+@DFExperimental
+public fun <T> ObservableMutableMeta.bind(
+    scope: CoroutineScope,
+    converter: MetaConverter<T>,
+    flow: MutableSharedFlow<T>,
+): Job = scope.launch{
+    listenTo(this, converter,flow)
+    onChange(flow){
+        launch {
+            flow.emit(converter.read(this@onChange))
+        }
+    }
+    flow.onCompletion {
+        removeListener(flow)
+    }
+}.also {
+    it.invokeOnCompletion {
+        removeListener(flow)
+    }
+}

dataforge-context/src/commonMain/kotlin/space/kscience/dataforge/properties/schemeProperty.kt

@@ -1,31 +0,0 @@
-package space.kscience.dataforge.properties
-
-
-import space.kscience.dataforge.meta.Scheme
-import space.kscience.dataforge.misc.DFExperimental
-import space.kscience.dataforge.names.parseAsName
-import space.kscience.dataforge.names.startsWith
-import kotlin.reflect.KMutableProperty1
-
-@DFExperimental
-public fun <S : Scheme, T : Any> S.property(property: KMutableProperty1<S, T?>): Property<T?> =
-    object : Property<T?> {
-        override var value: T?
-            get() = property.get(this@property)
-            set(value) {
-                property.set(this@property, value)
-            }
-
-        override fun onChange(owner: Any?, callback: (T?) -> Unit) {
-            this@property.meta.onChange(this) { name ->
-                if (name.startsWith(property.name.parseAsName(true))) {
-                    callback(property.get(this@property))
-                }
-            }
-        }
-
-        override fun removeChangeListener(owner: Any?) {
-            this@property.meta.removeListener(this@property)
-        }
-
-    }

dataforge-context/src/commonMain/kotlin/space/kscience/dataforge/provider/Path.kt

@@ -15,15 +15,37 @@
  */
 package space.kscience.dataforge.provider
 
+import kotlinx.serialization.KSerializer
+import kotlinx.serialization.Serializable
+import kotlinx.serialization.builtins.serializer
+import kotlinx.serialization.descriptors.SerialDescriptor
+import kotlinx.serialization.encoding.Decoder
+import kotlinx.serialization.encoding.Encoder
 import space.kscience.dataforge.names.Name
 import space.kscience.dataforge.names.parseAsName
 import kotlin.jvm.JvmInline
 
+private object PathSerializer : KSerializer<Path> {
+
+    override val descriptor: SerialDescriptor
+        get() = String.serializer().descriptor
+
+    override fun serialize(encoder: Encoder, value: Path) {
+        encoder.encodeString(value.toString())
+    }
+
+    override fun deserialize(decoder: Decoder): Path {
+        return Path.parse(decoder.decodeString())
+    }
+}
+
+
 /**
  * Path interface.
  *
  */
 @JvmInline
+@Serializable(PathSerializer::class)
 public value class Path(public val tokens: List<PathToken>) : Iterable<PathToken> {
 
     override fun iterator(): Iterator<PathToken> = tokens.iterator()
@@ -33,6 +55,7 @@ public value class Path(public val tokens: List<PathToken>) : Iterable<PathToken
     public companion object {
         public const val PATH_SEGMENT_SEPARATOR: String = "/"
 
+
         public fun parse(path: String): Path = Path(path.split(PATH_SEGMENT_SEPARATOR).map { PathToken.parse(it) })
     }
 }

dataforge-context/src/commonTest/kotlin/space/kscience/dataforge/properties/MetaPropertiesTest.kt

@@ -1,28 +0,0 @@
-package space.kscience.dataforge.properties
-
-import space.kscience.dataforge.meta.Scheme
-import space.kscience.dataforge.meta.SchemeSpec
-import space.kscience.dataforge.meta.int
-import space.kscience.dataforge.misc.DFExperimental
-import kotlin.test.Test
-import kotlin.test.assertEquals
-
-internal class TestScheme : Scheme() {
-    var a by int()
-    var b by int()
-    companion object : SchemeSpec<TestScheme>(::TestScheme)
-}
-
-@DFExperimental
-class MetaPropertiesTest {
-    @Test
-    fun testBinding() {
-        val scheme = TestScheme.empty()
-        val a = scheme.property(TestScheme::a)
-        val b = scheme.property(TestScheme::b)
-        a.bind(b)
-        scheme.a = 2
-        assertEquals(2, scheme.b)
-        assertEquals(2, b.value)
-    }
-}

dataforge-context/src/jsMain/kotlin/space/kscience/dataforge/properties/bindings.kt

@@ -1,32 +0,0 @@
-package space.kscience.dataforge.properties
-
-import org.w3c.dom.HTMLInputElement
-import space.kscience.dataforge.misc.DFExperimental
-
-@DFExperimental
-public fun HTMLInputElement.bindValue(property: Property<String>) {
-    if (this.onchange != null) error("Input element already bound")
-    this.onchange = {
-        property.value = this.value
-        Unit
-    }
-    property.onChange(this) {
-        if (value != it) {
-            value = it
-        }
-    }
-}
-
-@DFExperimental
-public fun HTMLInputElement.bindChecked(property: Property<Boolean>) {
-    if (this.onchange != null) error("Input element already bound")
-    this.onchange = {
-        property.value = this.checked
-        Unit
-    }
-    property.onChange(this) {
-        if (checked != it) {
-            checked = it
-        }
-    }
-}

dataforge-context/src/jvmMain/kotlin/space/kscience/dataforge/context/ClassLoaderPlugin.kt

@@ -17,7 +17,7 @@ package space.kscience.dataforge.context
 
 import java.util.*
 import kotlin.reflect.KClass
-import kotlin.reflect.full.cast
+import kotlin.reflect.cast
 
 public class ClassLoaderPlugin(private val classLoader: ClassLoader) : AbstractPlugin() {
     override val tag: PluginTag = PluginTag("classLoader", PluginTag.DATAFORGE_GROUP)

dataforge-context/src/jvmMain/kotlin/space/kscience/dataforge/descriptors/annotations.kt

@@ -1,126 +0,0 @@
-/*
- * Copyright  2018 Alexander Nozik.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *      http://www.apache.org/licenses/LICENSE-2.0
- *
- *  Unless required by applicable law or agreed to in writing, software
- *  distributed under the License is distributed on an "AS IS" BASIS,
- *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- *  See the License for the specific language governing permissions and
- *  limitations under the License.
- */
-
-package space.kscience.dataforge.descriptors
-
-//@MustBeDocumented
-//annotation class Attribute(
-//    val key: String,
-//    val value: String
-//)
-//
-//@MustBeDocumented
-//annotation class Attributes(
-//    val attrs: Array<Attribute>
-//)
-//
-//@MustBeDocumented
-//annotation class ItemDef(
-//    val info: String = "",
-//    val multiple: Boolean = false,
-//    val required: Boolean = false
-//)
-//
-//@Target(AnnotationTarget.PROPERTY)
-//@MustBeDocumented
-//annotation class ValueDef(
-//    val type: Array<ValueType> = [ValueType.STRING],
-//    val def: String = "",
-//    val allowed: Array<String> = [],
-//    val enumeration: KClass<*> = Any::class
-//)
-
-///**
-// * Description text for meta property, node or whole object
-// */
-//@Target(AnnotationTarget.CLASS, AnnotationTarget.PROPERTY)
-//@Retention(AnnotationRetention.RUNTIME)
-//@MustBeDocumented
-//annotation class Description(val value: String)
-//
-///**
-// * Annotation for value property which states that lists are expected
-// */
-//@Target(AnnotationTarget.PROPERTY)
-//@Retention(AnnotationRetention.RUNTIME)
-//@MustBeDocumented
-//annotation class Multiple
-//
-///**
-// * Descriptor target
-// * The DataForge path to the resource containing the description. Following targets are supported:
-// *  1. resource
-// *  1. file
-// *  1. class
-// *  1. method
-// *  1. property
-// *
-// *
-// * Does not work if [type] is provided
-// */
-//@Target(AnnotationTarget.CLASS, AnnotationTarget.PROPERTY, AnnotationTarget.VALUE_PARAMETER)
-//@Retention(AnnotationRetention.RUNTIME)
-//@MustBeDocumented
-//annotation class Descriptor(val value: String)
-//
-//
-///**
-// * Aggregator class for descriptor nodes
-// */
-//@Target(AnnotationTarget.CLASS, AnnotationTarget.PROPERTY, AnnotationTarget.FUNCTION, AnnotationTarget.VALUE_PARAMETER)
-//@Retention(AnnotationRetention.RUNTIME)
-//@MustBeDocumented
-//annotation class DescriptorNodes(vararg val nodes: NodeDef)
-//
-///**
-// * Aggregator class for descriptor values
-// */
-//@Target(AnnotationTarget.CLASS, AnnotationTarget.PROPERTY, AnnotationTarget.FUNCTION, AnnotationTarget.VALUE_PARAMETER)
-//@Retention(AnnotationRetention.RUNTIME)
-//@MustBeDocumented
-//annotation class DescriptorValues(vararg val nodes: ValueDef)
-//
-///**
-// * Alternative name for property descriptor declaration
-// */
-//@Target(AnnotationTarget.PROPERTY, AnnotationTarget.VALUE_PARAMETER)
-//@Retention(AnnotationRetention.RUNTIME)
-//@MustBeDocumented
-//annotation class DescriptorName(val name: String)
-//
-//@Target(AnnotationTarget.PROPERTY)
-//@Retention(AnnotationRetention.RUNTIME)
-//@MustBeDocumented
-//annotation class DescriptorValue(val def: ValueDef)
-////TODO enter fields directly?
-//
-//@Target(AnnotationTarget.PROPERTY)
-//@Retention(AnnotationRetention.RUNTIME)
-//@MustBeDocumented
-//annotation class ValueProperty(
-//    val name: String = "",
-//    val type: Array<ValueType> = arrayOf(ValueType.STRING),
-//    val multiple: Boolean = false,
-//    val def: String = "",
-//    val enumeration: KClass<*> = Any::class,
-//    val tags: Array<String> = emptyArray()
-//)
-//
-//
-//@Target(AnnotationTarget.PROPERTY)
-//@Retention(AnnotationRetention.RUNTIME)
-//@MustBeDocumented
-//annotation class NodeProperty(val name: String = "")

dataforge-context/src/jvmMain/kotlin/space/kscience/dataforge/descriptors/reflectiveDescriptors.kt

@@ -1,53 +1,132 @@
 package space.kscience.dataforge.descriptors
 
+import kotlinx.serialization.ExperimentalSerializationApi
+import kotlinx.serialization.json.Json
+import kotlinx.serialization.json.decodeFromStream
+import org.slf4j.LoggerFactory
+import space.kscience.dataforge.meta.descriptors.MetaDescriptor
+import space.kscience.dataforge.meta.descriptors.MetaDescriptorBuilder
+import space.kscience.dataforge.misc.DFExperimental
+import java.net.URL
+import kotlin.reflect.KAnnotatedElement
+import kotlin.reflect.KProperty
 
-//inline fun <reified T : Scheme> T.buildDescriptor(): NodeDescriptor = NodeDescriptor {
+/**
-//    T::class.apply {
+ * Description text for meta property, node or whole object
-//        findAnnotation<ItemDef>()?.let { def ->
+ */
-//            info = def.info
+@Target(AnnotationTarget.CLASS, AnnotationTarget.PROPERTY)
-//            required = def.required
+@Retention(AnnotationRetention.RUNTIME)
-//            multiple = def.multiple
+@MustBeDocumented
-//        }
+public annotation class Description(val value: String)
-//        findAnnotation<Attribute>()?.let { attr ->
+
-//            attributes {
+@Target(AnnotationTarget.PROPERTY)
-//                this[attr.key] = attr.value.parseValue()
+@Retention(AnnotationRetention.RUNTIME)
-//            }
+@MustBeDocumented
-//        }
+public annotation class Multiple()
-//        findAnnotation<Attributes>()?.attrs?.forEach { attr ->
+
-//            attributes {
+@Target(AnnotationTarget.CLASS, AnnotationTarget.PROPERTY)
-//                this[attr.key] = attr.value.parseValue()
+@Retention(AnnotationRetention.RUNTIME)
-//            }
+@MustBeDocumented
-//        }
+public annotation class DescriptorResource(val resourceName: String)
-//    }
+
-//    T::class.memberProperties.forEach { property ->
+@Target(AnnotationTarget.CLASS, AnnotationTarget.PROPERTY)
-//        val delegate = property.getDelegate(this@buildDescriptor)
+@Retention(AnnotationRetention.RUNTIME)
+@MustBeDocumented
+public annotation class DescriptorUrl(val url: String)
+
+
+@OptIn(ExperimentalSerializationApi::class)
+private fun MetaDescriptorBuilder.loadDescriptorFromUrl(url: URL) {
+    url.openStream().use {
+        from(Json.decodeFromStream(MetaDescriptor.serializer(), it))
+    }
+}
+
+private fun MetaDescriptorBuilder.loadDescriptorFromResource(resource: DescriptorResource) {
+    val url = {}.javaClass.getResource(resource.resourceName)
+    if (url != null) {
+        loadDescriptorFromUrl(url)
+    } else {
+        LoggerFactory.getLogger("System")
+            .error("Can't find descriptor resource with name ${resource.resourceName}")
+    }
+}
+
+@DFExperimental
+public fun MetaDescriptorBuilder.forAnnotatedElement(element: KAnnotatedElement) {
+    element.annotations.forEach {
+        when (it) {
+            is Description -> description = it.value
+
+            is DescriptorResource -> loadDescriptorFromResource(it)
+
+            is DescriptorUrl -> loadDescriptorFromUrl(URL(it.url))
+        }
+    }
+}
+
+@DFExperimental
+public fun MetaDescriptorBuilder.forProperty(property: KProperty<*>) {
+    property.annotations.forEach {
+        when (it) {
+            is Description -> description = it.value
+
+            is DescriptorResource -> loadDescriptorFromResource(it)
+
+            is DescriptorUrl -> loadDescriptorFromUrl(URL(it.url))
+        }
+    }
+}
 //
-//        val descriptor: ItemDescriptor = when (delegate) {
+//@DFExperimental
-//            is ConfigurableDelegate -> buildPropertyDescriptor(property, delegate)
+//public fun <T : Scheme> MetaDescriptor.Companion.forScheme(
-//            is ReadWriteDelegateWrapper<*, *> -> {
+//    spec: SchemeSpec<T>,
-//                if (delegate.delegate is ConfigurableDelegate) {
+//    mod: MetaDescriptorBuilder.() -> Unit = {},
-//                    buildPropertyDescriptor(property, delegate.delegate as ConfigurableDelegate)
+//): MetaDescriptor = MetaDescriptor {
-//                } else {
+//    val scheme = spec.empty()
-//                    return@forEach
+//    val kClass: KClass<T> = scheme::class as KClass<T>
+//    when {
+//        kClass.isSubclassOf(Number::class) -> valueType(ValueType.NUMBER)
+//        kClass == String::class -> ValueType.STRING
+//        kClass == Boolean::class -> ValueType.BOOLEAN
+//        kClass == DoubleArray::class -> ValueType.LIST
+//        kClass == ByteArray::class -> ValueType.LIST
+//    }
+//
+//    forAnnotatedElement(kClass)
+//    kClass.memberProperties.forEach { property ->
+//        node(property.name) {
+//
+//            (property.getDelegate(scheme) as? MetaDelegate<*>)?.descriptor?.let {
+//                from(it)
+//            }
+//
+//            property.annotations.forEach {
+//                when (it) {
+//                    is Description -> {
+//                        description = it.value
+//                    }
+//
+//                    is Multiple -> {
+//                        multiple = true
+//                    }
+//
+//                    is DescriptorResource -> {
+//                        loadDescriptorFromResource(it)
+//                    }
+//
+//                    is DescriptorUrl -> {
+//                        loadDescriptorFromUrl(URL(it.url))
+//                    }
 //                }
 //            }
-//            else -> return@forEach
-//        }
-//        defineItem(property.name, descriptor)
-//    }
-//}
-
-//inline fun <T : Scheme, reified V : Any?> buildPropertyDescriptor(
-//    property: KProperty1<T, V>,
-//    delegate: ConfigurableDelegate
-//): ItemDescriptor {
-//    when {
-//        V::class.isSubclassOf(Scheme::class) -> NodeDescriptor {
-//            default = delegate.default.node
-//        }
-//        V::class.isSubclassOf(Meta::class) -> NodeDescriptor {
-//            default = delegate.default.node
 //        }
 //
 //    }
+//    mod()
 //}
+//
+//@DFExperimental
+//public inline fun <reified T : Scheme> SchemeSpec<T>.autoDescriptor(
+//    noinline mod: MetaDescriptorBuilder.() -> Unit = {},
+//): MetaDescriptor = MetaDescriptor.forScheme(this, mod)

dataforge-context/src/jvmMain/kotlin/space/kscience/dataforge/provider/dfType.kt

@@ -4,29 +4,34 @@ import space.kscience.dataforge.context.Context
 import space.kscience.dataforge.context.PluginBuilder
 import space.kscience.dataforge.context.gather
 import space.kscience.dataforge.misc.DFExperimental
-import space.kscience.dataforge.misc.DfId
+import space.kscience.dataforge.misc.DfType
 import space.kscience.dataforge.misc.Named
 import space.kscience.dataforge.names.Name
 import kotlin.reflect.KClass
+import kotlin.reflect.KType
 import kotlin.reflect.full.findAnnotation
-
+import kotlin.reflect.typeOf
 
 @DFExperimental
-public val KClass<*>.dfId: String
+public val KClass<*>.dfType: String
-    get() = findAnnotation<DfId>()?.id ?: simpleName ?: ""
+    get() = findAnnotation<DfType>()?.id ?: simpleName ?: ""
+
+@DFExperimental
+public val KType.dfType: String
+    get() = findAnnotation<DfType>()?.id ?: (classifier as? KClass<*>)?.simpleName ?: ""
 
 /**
- * Provide an object with given name inferring target from its type using [DfId] annotation
+ * Provide an object with given name inferring target from its type using [DfType] annotation
  */
 @DFExperimental
 public inline fun <reified T : Any> Provider.provideByType(name: String): T? {
-    val target = T::class.dfId
+    val target = typeOf<T>().dfType
     return provide(target, name)
 }
 
 @DFExperimental
 public inline fun <reified T : Any> Provider.top(): Map<Name, T> {
-    val target = T::class.dfId
+    val target = typeOf<T>().dfType
     return top(target)
 }
 
@@ -35,15 +40,15 @@ public inline fun <reified T : Any> Provider.top(): Map<Name, T> {
  */
 @DFExperimental
 public inline fun <reified T : Any> Context.gather(inherit: Boolean = true): Map<Name, T> =
-    gather<T>(T::class.dfId, inherit)
+    gather<T>(typeOf<T>().dfType, inherit)
 
 
 @DFExperimental
 public inline fun <reified T : Any> PluginBuilder.provides(items: Map<Name, T>) {
-    provides(T::class.dfId, items)
+    provides(typeOf<T>().dfType, items)
 }
 
 @DFExperimental
 public inline fun <reified T : Any> PluginBuilder.provides(vararg items: Named) {
-    provides(T::class.dfId, *items)
+    provides(typeOf<T>().dfType, *items)
 }

dataforge-context/src/jvmTest/kotlin/space/kscience/dataforge/descriptors/TestAutoDescriptors.kt

@@ -0,0 +1,29 @@
+@file:OptIn(DFExperimental::class)
+
+package space.kscience.dataforge.descriptors
+
+import space.kscience.dataforge.misc.DFExperimental
+
+//
+//class TestScheme : Scheme() {
+//
+//    @Description("A")
+//    val a by string()
+//
+//    @Description("B")
+//    val b by int()
+//
+//    val c by int()
+//
+//    companion object : SchemeSpec<TestScheme>(::TestScheme) {
+//        override val descriptor: MetaDescriptor = autoDescriptor()
+//    }
+//}
+//
+//class TestAutoDescriptors {
+//    @Test
+//    fun autoDescriptor() {
+//        val autoDescriptor = MetaDescriptor.forScheme(TestScheme)
+//        println(Json { prettyPrint = true }.encodeToString(autoDescriptor))
+//    }
+//}

dataforge-context/src/wasmJsMain/kotlin/space/kscience/dataforge/context/loggingWasm.kt

@@ -0,0 +1,3 @@
+package space.kscience.dataforge.context
+
+internal actual fun getGlobalLoggerFactory(): PluginFactory<out LogManager> = DefaultLogManager

dataforge-data/README.md

@@ -6,18 +6,16 @@
 
 ## Artifact:
 
-The Maven coordinates of this project are `space.kscience:dataforge-data:0.7.0`.
+The Maven coordinates of this project are `space.kscience:dataforge-data:0.10.0`.
 
 **Gradle Kotlin DSL:**
 ```kotlin
 repositories {
     maven("https://repo.kotlin.link")
-    //uncomment to access development builds
-    //maven("https://maven.pkg.jetbrains.space/spc/p/sci/dev")
     mavenCentral()
 }
 
 dependencies {
-    implementation("space.kscience:dataforge-data:0.7.0")
+    implementation("space.kscience:dataforge-data:0.10.0")
 }
 ```

dataforge-data/build.gradle.kts

@@ -6,10 +6,10 @@ kscience{
     jvm()
     js()
     native()
+    wasm()
     useCoroutines()
     dependencies {
-        api(project(":dataforge-meta"))
+        api(projects.dataforgeMeta)
-        api(kotlin("reflect"))
     }
 }
 

dataforge-data/src/commonMain/kotlin/space/kscience/dataforge/actions/AbstractAction.kt

@@ -1,9 +1,9 @@
 package space.kscience.dataforge.actions
 
-import kotlinx.coroutines.launch
+import kotlinx.coroutines.CoroutineScope
 import space.kscience.dataforge.data.*
 import space.kscience.dataforge.meta.Meta
-import space.kscience.dataforge.misc.DFInternal
+import space.kscience.dataforge.misc.UnsafeKType
 import space.kscience.dataforge.names.Name
 import space.kscience.dataforge.names.startsWith
 import kotlin.reflect.KType
@@ -19,47 +19,55 @@ internal fun MutableMap<Name, *>.removeWhatStartsWith(name: Name) {
 /**
  * An action that caches results on-demand and recalculates them on source push
  */
-public abstract class AbstractAction<in T : Any, R : Any>(
+public abstract class AbstractAction<T, R>(
     public val outputType: KType,
 ) : Action<T, R> {
 
     /**
      * Generate initial content of the output
      */
-    protected abstract fun DataSetBuilder<R>.generate(
+    protected abstract fun DataBuilderScope<R>.generate(
-        data: DataSet<T>,
+        source: DataTree<T>,
         meta: Meta,
-    )
+    ): Map<Name, Data<R>>
 
     /**
-     * Update part of the data set when given [updateKey] is triggered by the source
+     * Update part of the data set using provided data
+     *
+     * @param source the source data tree in case we need several data items to update
+     * @param actionMeta the metadata used for the whole data tree
+     * @param updatedData an updated item
      */
-    protected open fun DataSourceBuilder<R>.update(
+    protected open suspend fun DataSink<R>.update(
-        dataSet: DataSet<T>,
+        source: DataTree<T>,
-        meta: Meta,
+        actionMeta: Meta,
-        updateKey: Name,
+        updateName: Name,
     ) {
-        // By default, recalculate the whole dataset
+        //by default regenerate the whole data set
-        generate(dataSet, meta)
+        writeAll(generate(source, actionMeta))
     }
 
-    @OptIn(DFInternal::class)
+    @OptIn(UnsafeKType::class)
     override fun execute(
-        dataSet: DataSet<T>,
+        source: DataTree<T>,
         meta: Meta,
-    ): DataSet<R> = if (dataSet is DataSource) {
+        updatesScope: CoroutineScope
-        DataSource(outputType, dataSet){
+    ): DataTree<R> = DataTree(
-            generate(dataSet, meta)
+        dataType = outputType,
+        scope = updatesScope,
+        initialData = DataBuilderScope<R>().generate(source, meta)
+    ) {
 
-            launch {
+        //propagate updates
-                dataSet.updates.collect { name ->
+        val updateSink = DataSink<R> { name, data ->
-                    update(dataSet, meta, name)
+            write(name, data)
-                }
-            }
         }
-    } else {
+
-        DataTree<R>(outputType) {
+        with(updateSink) {
-            generate(dataSet, meta)
+            source.updates.collect {
+                update(source, meta, it)
+            }
         }
     }
 }
+

dataforge-data/src/commonMain/kotlin/space/kscience/dataforge/actions/Action.kt

@@ -1,40 +1,50 @@
 package space.kscience.dataforge.actions
 
-import space.kscience.dataforge.data.DataSet
+import kotlinx.coroutines.CoroutineScope
+import kotlinx.coroutines.DelicateCoroutinesApi
+import kotlinx.coroutines.GlobalScope
+import space.kscience.dataforge.data.DataTree
 import space.kscience.dataforge.meta.Meta
 import space.kscience.dataforge.misc.DFExperimental
 
 /**
  * A simple data transformation on a data node. Actions should avoid doing actual dependency evaluation in [execute].
  */
-public interface Action<in T : Any, out R : Any> {
+public fun interface Action<T, R> {
 
     /**
      * Transform the data in the node, producing a new node. By default, it is assumed that all calculations are lazy
      * so not actual computation is started at this moment.
      */
-    public fun execute(dataSet: DataSet<T>, meta: Meta = Meta.EMPTY): DataSet<R>
+    public fun execute(source: DataTree<T>, meta: Meta, updatesScope: CoroutineScope): DataTree<R>
 
     public companion object
 }
 
+/**
+ * A convenience method to transform data using given [action]
+ */
+@OptIn(DelicateCoroutinesApi::class)
+public fun <T, R> DataTree<T>.transform(
+    action: Action<T, R>,
+    meta: Meta = Meta.EMPTY,
+    updateScope: CoroutineScope = GlobalScope,
+): DataTree<R> = action.execute(this, meta, updateScope)
+
 /**
  * Action composition. The result is terminal if one of its parts is terminal
  */
-public infix fun <T : Any, I : Any, R : Any> Action<T, I>.then(action: Action<I, R>): Action<T, R> {
+public infix fun <T, I, R> Action<T, I>.then(action: Action<I, R>): Action<T, R> = Action { dataSet, meta, scope ->
-    // TODO introduce composite action and add optimize by adding action to the list
+    action.execute(this@then.execute(dataSet, meta, scope), meta, scope)
-    return object : Action<T, R> {
-
-        override fun execute(
-            dataSet: DataSet<T>,
-            meta: Meta,
-        ): DataSet<R> = action.execute(this@then.execute(dataSet, meta), meta)
-    }
 }
 
 @DFExperimental
-public operator fun <T : Any, R : Any> Action<T, R>.invoke(
+@OptIn(DelicateCoroutinesApi::class)
-    dataSet: DataSet<T>,
+public operator fun <T, R> Action<T, R>.invoke(
+    dataSet: DataTree<T>,
     meta: Meta = Meta.EMPTY,
-): DataSet<R> = execute(dataSet, meta)
+    updateScope: CoroutineScope = GlobalScope,
+): DataTree<R> = execute(dataSet, meta, updateScope)
+
+
 

dataforge-data/src/commonMain/kotlin/space/kscience/dataforge/actions/MapAction.kt

@@ -6,8 +6,7 @@ import space.kscience.dataforge.meta.MutableMeta
 import space.kscience.dataforge.meta.seal
 import space.kscience.dataforge.meta.toMutableMeta
 import space.kscience.dataforge.misc.DFBuilder
-import space.kscience.dataforge.misc.DFExperimental
+import space.kscience.dataforge.misc.UnsafeKType
-import space.kscience.dataforge.misc.DFInternal
 import space.kscience.dataforge.names.Name
 import kotlin.reflect.KType
 import kotlin.reflect.typeOf
@@ -29,6 +28,7 @@ public class MapActionBuilder<T, R>(
     public var name: Name,
     public var meta: MutableMeta,
     public val actionMeta: Meta,
+    public val dataType: KType,
     @PublishedApi internal var outputType: KType,
 ) {
 
@@ -37,6 +37,7 @@ public class MapActionBuilder<T, R>(
     /**
      * Set unsafe [outputType] for the resulting data. Be sure that it is correct.
      */
+    @UnsafeKType
     public fun <R1 : R> result(outputType: KType, f: suspend ActionEnv.(T) -> R1) {
         this.outputType = outputType
         result = f;
@@ -45,28 +46,31 @@ public class MapActionBuilder<T, R>(
     /**
      * Calculate the result of goal
      */
-    public inline fun <reified R1 : R> result(noinline f: suspend ActionEnv.(T) -> R1) {
+    @OptIn(UnsafeKType::class)
-        outputType = typeOf<R1>()
+    public inline fun <reified R1 : R> result(noinline f: suspend ActionEnv.(T) -> R1): Unit = result(typeOf<R1>(), f)
-        result = f;
-    }
 }
 
-@PublishedApi
+@UnsafeKType
-internal class MapAction<in T : Any, R : Any>(
+public class MapAction<T, R>(
     outputType: KType,
     private val block: MapActionBuilder<T, R>.() -> Unit,
 ) : AbstractAction<T, R>(outputType) {
 
-    private fun DataSetBuilder<R>.mapOne(name: Name, data: Data<T>, meta: Meta) {
+    private fun mapOne(name: Name, data: Data<T>?, meta: Meta): Pair<Name, Data<R>?> {
+        //fast return for null data
+        if (data == null) {
+            return name to null
+        }
         // Creating a new environment for action using **old** name, old meta and task meta
         val env = ActionEnv(name, data.meta, meta)
 
         //applying transformation from builder
         val builder = MapActionBuilder<T, R>(
-            name,
+            name = name,
-            data.meta.toMutableMeta(), // using data meta
+            meta = data.meta.toMutableMeta(), // using data meta
-            meta,
+            actionMeta = meta,
-            outputType
+            dataType = data.type,
+            outputType = outputType
         ).apply(block)
 
         //getting new name
@@ -75,21 +79,30 @@ internal class MapAction<in T : Any, R : Any>(
         //getting new meta
         val newMeta = builder.meta.seal()
 
-        @OptIn(DFInternal::class)
         val newData = Data(builder.outputType, newMeta, dependencies = listOf(data)) {
             builder.result(env, data.await())
         }
         //setting the data node
-        data(newName, newData)
+        return newName to newData
     }
 
-    override fun DataSetBuilder<R>.generate(data: DataSet<T>, meta: Meta) {
+    override fun DataBuilderScope<R>.generate(source: DataTree<T>, meta: Meta): Map<Name, Data<R>> = buildMap {
-        data.forEach { mapOne(it.name, it.data, meta) }
+        source.forEach { data ->
+            val (name, data) = mapOne(data.name, data, meta)
+            if (data != null) {
+                check(name !in keys) { "Data with key $name already exist in the result" }
+                put(name, data)
+            }
+        }
     }
 
-    override fun DataSourceBuilder<R>.update(dataSet: DataSet<T>, meta: Meta, updateKey: Name) {
+    override suspend fun DataSink<R>.update(
-        remove(updateKey)
+        source: DataTree<T>,
-        dataSet[updateKey]?.let { mapOne(updateKey, it, meta) }
+        actionMeta: Meta,
+        updateName: Name,
+    ) {
+        val (name, data) = mapOne(updateName, source.read(updateName), actionMeta)
+        write(name, data)
     }
 }
 
@@ -97,9 +110,9 @@ internal class MapAction<in T : Any, R : Any>(
 /**
  * A one-to-one mapping action
  */
-@DFExperimental
+
-@Suppress("FunctionName")
+@OptIn(UnsafeKType::class)
-public inline fun <T : Any, reified R : Any> Action.Companion.map(
+public inline fun <T, reified R> Action.Companion.mapping(
     noinline builder: MapActionBuilder<T, R>.() -> Unit,
 ): Action<T, R> = MapAction(typeOf<R>(), builder)
 

dataforge-data/src/commonMain/kotlin/space/kscience/dataforge/actions/ReduceAction.kt

@@ -3,18 +3,19 @@ package space.kscience.dataforge.actions
 import space.kscience.dataforge.data.*
 import space.kscience.dataforge.meta.Meta
 import space.kscience.dataforge.meta.MutableMeta
+import space.kscience.dataforge.meta.get
+import space.kscience.dataforge.meta.string
 import space.kscience.dataforge.misc.DFBuilder
-import space.kscience.dataforge.misc.DFExperimental
+import space.kscience.dataforge.misc.UnsafeKType
-import space.kscience.dataforge.misc.DFInternal
 import space.kscience.dataforge.names.Name
 import space.kscience.dataforge.names.parseAsName
 import kotlin.reflect.KType
 import kotlin.reflect.typeOf
 
 
-public class JoinGroup<T : Any, R : Any>(
+public class JoinGroup<T, R>(
     public var name: String,
-    internal val set: DataSet<T>,
+    internal val data: DataTree<T>,
     @PublishedApi internal var outputType: KType,
 ) {
 
@@ -22,12 +23,12 @@ public class JoinGroup<T : Any, R : Any>(
 
     public lateinit var result: suspend ActionEnv.(Map<Name, ValueWithMeta<T>>) -> R
 
-    internal fun <R1 : R> result(outputType: KType, f: suspend ActionEnv.(Map<Name,  ValueWithMeta<T>>) -> R1) {
+    internal fun <R1 : R> result(outputType: KType, f: suspend ActionEnv.(Map<Name, ValueWithMeta<T>>) -> R1) {
         this.outputType = outputType
         this.result = f;
     }
 
-    public inline fun <reified R1 : R> result(noinline f: suspend ActionEnv.(Map<Name,  ValueWithMeta<T>>) -> R1) {
+    public inline fun <reified R1 : R> result(noinline f: suspend ActionEnv.(Map<Name, ValueWithMeta<T>>) -> R1) {
         outputType = typeOf<R1>()
         this.result = f;
     }
@@ -35,31 +36,36 @@ public class JoinGroup<T : Any, R : Any>(
 }
 
 @DFBuilder
-public class ReduceGroupBuilder<T : Any, R : Any>(
+public class ReduceGroupBuilder<T, R>(
     public val actionMeta: Meta,
     private val outputType: KType,
 ) {
-    private val groupRules: MutableList<(DataSet<T>) -> List<JoinGroup<T, R>>> = ArrayList();
+    private val groupRules: MutableList<(DataTree<T>) -> List<JoinGroup<T, R>>> = ArrayList();
 
     /**
-     * introduce grouping by meta value
+     * Group by a meta value
      */
-    public fun byValue(tag: String, defaultTag: String = "@default", action: JoinGroup<T, R>.() -> Unit) {
+    @OptIn(UnsafeKType::class)
+    public fun byMetaValue(tag: String, defaultTag: String = "@default", action: JoinGroup<T, R>.() -> Unit) {
         groupRules += { node ->
-            GroupRule.byMetaValue(tag, defaultTag).gather(node).map {
+            val groups = mutableMapOf<String, MutableMap<Name, Data<T>>>()
-                JoinGroup<T, R>(it.key, it.value, outputType).apply(action)
+            node.forEach { data ->
+                groups.getOrPut(data.meta[tag]?.string ?: defaultTag) { mutableMapOf() }.put(data.name, data)
+            }
+            groups.map { (key, dataMap) ->
+                JoinGroup<T, R>(key, dataMap.asTree(node.dataType), outputType).apply(action)
             }
         }
     }
 
     public fun group(
         groupName: String,
-        predicate: (Name, Meta) -> Boolean,
+        predicate: DataFilter,
         action: JoinGroup<T, R>.() -> Unit,
     ) {
         groupRules += { source ->
             listOf(
-                JoinGroup<T, R>(groupName, source.filter(predicate), outputType).apply(action)
+                JoinGroup<T, R>(groupName, source.filterData(predicate), outputType).apply(action)
             )
         }
     }
@@ -67,29 +73,29 @@ public class ReduceGroupBuilder<T : Any, R : Any>(
     /**
      * Apply transformation to the whole node
      */
-    public fun result(resultName: String, f: suspend ActionEnv.(Map<Name,  ValueWithMeta<T>>) -> R) {
+    public fun result(resultName: String, f: suspend ActionEnv.(Map<Name, ValueWithMeta<T>>) -> R) {
         groupRules += { node ->
             listOf(JoinGroup<T, R>(resultName, node, outputType).apply { result(outputType, f) })
         }
     }
 
-    internal fun buildGroups(input: DataSet<T>): List<JoinGroup<T, R>> =
+    internal fun buildGroups(input: DataTree<T>): List<JoinGroup<T, R>> =
         groupRules.flatMap { it.invoke(input) }
 
 }
 
 @PublishedApi
-internal class ReduceAction<T : Any, R : Any>(
+internal class ReduceAction<T, R>(
     outputType: KType,
     private val action: ReduceGroupBuilder<T, R>.() -> Unit,
 ) : AbstractAction<T, R>(outputType) {
     //TODO optimize reduction. Currently, the whole action recalculates on push
 
-    override fun DataSetBuilder<R>.generate(data: DataSet<T>, meta: Meta) {
+    override fun DataBuilderScope<R>.generate(source: DataTree<T>, meta: Meta): Map<Name, Data<R>> = buildMap {
-        ReduceGroupBuilder<T, R>(meta, outputType).apply(action).buildGroups(data).forEach { group ->
+        ReduceGroupBuilder<T, R>(meta, outputType).apply(action).buildGroups(source).forEach { group ->
-            val dataFlow: Map<Name, Data<T>> = group.set.asSequence().fold(HashMap()) { acc, value ->
+            val dataFlow: Map<Name, Data<T>> = group.data.asSequence().fold(HashMap()) { acc, value ->
                 acc.apply {
-                    acc[value.name] = value.data
+                    acc[value.name] = value
                 }
             }
 
@@ -98,12 +104,12 @@ internal class ReduceAction<T : Any, R : Any>(
             val groupMeta = group.meta
 
             val env = ActionEnv(groupName.parseAsName(), groupMeta, meta)
-            @OptIn(DFInternal::class) val res: Data<R> = dataFlow.reduceToData(
+            @OptIn(UnsafeKType::class) val res: Data<R> = dataFlow.reduceToData(
                 group.outputType,
                 meta = groupMeta
             ) { group.result.invoke(env, it) }
 
-            data(env.name, res)
+            put(env.name, res)
         }
     }
 }
@@ -111,7 +117,6 @@ internal class ReduceAction<T : Any, R : Any>(
 /**
  * A one-to-one mapping action
  */
-@DFExperimental
+public inline fun <reified T, reified R> Action.Companion.reducing(
-public inline fun <reified T : Any, reified R : Any> Action.Companion.reduce(
     noinline builder: ReduceGroupBuilder<T, R>.() -> Unit,
 ): Action<T, R> = ReduceAction(typeOf<R>(), builder)

dataforge-data/src/commonMain/kotlin/space/kscience/dataforge/actions/SplitAction.kt

@@ -5,17 +5,15 @@ import space.kscience.dataforge.meta.Laminate
 import space.kscience.dataforge.meta.Meta
 import space.kscience.dataforge.meta.MutableMeta
 import space.kscience.dataforge.meta.toMutableMeta
-import space.kscience.dataforge.misc.DFExperimental
 import space.kscience.dataforge.names.Name
 import space.kscience.dataforge.names.parseAsName
-import kotlin.collections.set
 import kotlin.reflect.KType
 import kotlin.reflect.typeOf
 
 
-public class SplitBuilder<T : Any, R : Any>(public val name: Name, public val meta: Meta) {
+public class SplitBuilder<T, R>(public val name: Name, public val meta: Meta) {
 
-    public class FragmentRule<T : Any, R : Any>(
+    public class FragmentRule<T, R>(
         public val name: Name,
         public var meta: MutableMeta,
         @PublishedApi internal var outputType: KType,
@@ -44,15 +42,15 @@ public class SplitBuilder<T : Any, R : Any>(public val name: Name, public val me
  * Action that splits each incoming element into a number of fragments defined in builder
  */
 @PublishedApi
-internal class SplitAction<T : Any, R : Any>(
+internal class SplitAction<T, R>(
     outputType: KType,
     private val action: SplitBuilder<T, R>.() -> Unit,
 ) : AbstractAction<T, R>(outputType) {
 
-    private fun DataSetBuilder<R>.splitOne(name: Name, data: Data<T>, meta: Meta) {
+    private fun splitOne(name: Name, data: Data<T>?, meta: Meta): Map<Name, Data<R>?> = buildMap {
-        val laminate = Laminate(data.meta, meta)
+        val laminate = Laminate(data?.meta, meta)
 
-        val split = SplitBuilder<T, R>(name, data.meta).apply(action)
+        val split = SplitBuilder<T, R>(name, data?.meta ?: Meta.EMPTY).apply(action)
 
 
         // apply individual fragment rules to result
@@ -64,29 +62,46 @@ internal class SplitAction<T : Any, R : Any>(
             ).apply(rule)
             //data.map<R>(outputType, meta = env.meta) { env.result(it) }.named(fragmentName)
 
-            data(
+            if (data == null) {
-                fragmentName,
+                put(fragmentName, null)
-                @Suppress("OPT_IN_USAGE") Data(outputType, meta = env.meta, dependencies = listOf(data)) {
+            } else {
-                    env.result(data.await())
+                put(
-                }
+                    fragmentName,
-            )
+                    @Suppress("OPT_IN_USAGE") Data(outputType, meta = env.meta, dependencies = listOf(data)) {
+                        env.result(data.await())
+                    }
+                )
+            }
         }
     }
 
-    override fun DataSetBuilder<R>.generate(data: DataSet<T>, meta: Meta) {
+    override fun DataBuilderScope<R>.generate(
-        data.forEach { splitOne(it.name, it.data, meta) }
+        source: DataTree<T>,
+        meta: Meta
+    ): Map<Name, Data<R>> = buildMap {
+        source.forEach {
+            splitOne(it.name, it, meta).forEach { (name, data) ->
+                check(name !in keys) { "Data with key $name already exist in the result" }
+                if (data != null) {
+                    put(name, data)
+                }
+            }
+        }
     }
 
-    override fun DataSourceBuilder<R>.update(dataSet: DataSet<T>, meta: Meta, updateKey: Name) {
+    override suspend fun DataSink<R>.update(
-        remove(updateKey)
+        source: DataTree<T>,
-        dataSet[updateKey]?.let { splitOne(updateKey, it, meta) }
+        actionMeta: Meta,
+        updateName: Name,
+    ) {
+        writeAll(splitOne(updateName, source.read(updateName), actionMeta))
     }
 }
 
 /**
  * Action that splits each incoming element into a number of fragments defined in builder
  */
-@DFExperimental
+
-public inline fun <T : Any, reified R : Any> Action.Companion.split(
+public inline fun <T, reified R> Action.Companion.splitting(
     noinline builder: SplitBuilder<T, R>.() -> Unit,
 ): Action<T, R> = SplitAction(typeOf<R>(), builder)

dataforge-data/src/commonMain/kotlin/space/kscience/dataforge/data/Data.kt

@@ -4,8 +4,8 @@ import kotlinx.coroutines.*
 import space.kscience.dataforge.meta.Meta
 import space.kscience.dataforge.meta.MetaRepr
 import space.kscience.dataforge.meta.isEmpty
-import space.kscience.dataforge.misc.DFInternal
+import space.kscience.dataforge.misc.DfType
-import space.kscience.dataforge.misc.DfId
+import space.kscience.dataforge.misc.UnsafeKType
 import kotlin.coroutines.CoroutineContext
 import kotlin.coroutines.EmptyCoroutineContext
 import kotlin.reflect.KType
@@ -14,7 +14,7 @@ import kotlin.reflect.typeOf
 /**
  * A data element characterized by its meta
  */
-@DfId(Data.TYPE)
+@DfType(Data.TYPE)
 public interface Data<out T> : Goal<T>, MetaRepr {
     /**
      * Type marker for the data. The type is known before the calculation takes place so it could be checked.
@@ -41,7 +41,7 @@ public interface Data<out T> : Goal<T>, MetaRepr {
          */
         internal val TYPE_OF_NOTHING: KType = typeOf<Unit>()
 
-        public inline fun <reified T : Any> static(
+        public inline fun <reified T> wrapValue(
             value: T,
             meta: Meta = Meta.EMPTY,
         ): Data<T> = StaticData(typeOf<T>(), value, meta)
@@ -50,10 +50,10 @@ public interface Data<out T> : Goal<T>, MetaRepr {
          * An empty data containing only meta
          */
         @OptIn(DelicateCoroutinesApi::class)
-        public fun empty(meta: Meta): Data<Nothing> = object : Data<Nothing> {
+        public fun buildEmpty(meta: Meta): Data<Nothing> = object : Data<Nothing> {
-            override val type: KType = TYPE_OF_NOTHING
+            override val type: KType get() = TYPE_OF_NOTHING
             override val meta: Meta = meta
-            override val dependencies: Collection<Goal<*>> = emptyList()
+            override val dependencies: Collection<Goal<*>> get() = emptyList()
             override val deferred: Deferred<Nothing>
                 get() = GlobalScope.async(start = CoroutineStart.LAZY) {
                     error("The Data is empty and could not be computed")
@@ -62,6 +62,8 @@ public interface Data<out T> : Goal<T>, MetaRepr {
             override fun async(coroutineScope: CoroutineScope): Deferred<Nothing> = deferred
             override fun reset() {}
         }
+
+        public val EMPTY: Data<Nothing> = buildEmpty(Meta.EMPTY)
     }
 }
 
@@ -69,39 +71,37 @@ public interface Data<out T> : Goal<T>, MetaRepr {
  * A lazily computed variant of [Data] based on [LazyGoal]
  * One must ensure that proper [type] is used so this method should not be used
  */
-private class LazyData<T : Any>(
+private class LazyData<T>(
     override val type: KType,
     override val meta: Meta = Meta.EMPTY,
     additionalContext: CoroutineContext = EmptyCoroutineContext,
-    dependencies: Collection<Goal<*>> = emptyList(),
+    dependencies: Iterable<Goal<*>> = emptyList(),
     block: suspend () -> T,
 ) : Data<T>, LazyGoal<T>(additionalContext, dependencies, block)
 
-public class StaticData<T : Any>(
+public class StaticData<T>(
     override val type: KType,
     value: T,
     override val meta: Meta = Meta.EMPTY,
 ) : Data<T>, StaticGoal<T>(value)
 
 @Suppress("FunctionName")
-public inline fun <reified T : Any> Data(value: T, meta: Meta = Meta.EMPTY): StaticData<T> =
+public inline fun <reified T> Data(value: T, meta: Meta = Meta.EMPTY): StaticData<T> =
     StaticData(typeOf<T>(), value, meta)
 
-@Suppress("FunctionName")
+@UnsafeKType
-@DFInternal
+public fun <T> Data(
-public fun <T : Any> Data(
     type: KType,
     meta: Meta = Meta.EMPTY,
     context: CoroutineContext = EmptyCoroutineContext,
-    dependencies: Collection<Goal<*>> = emptyList(),
+    dependencies: Iterable<Goal<*>> = emptyList(),
     block: suspend () -> T,
 ): Data<T> = LazyData(type, meta, context, dependencies, block)
 
-@OptIn(DFInternal::class)
+@OptIn(UnsafeKType::class)
-@Suppress("FunctionName")
+public inline fun <reified T> Data(
-public inline fun <reified T : Any> Data(
     meta: Meta = Meta.EMPTY,
     context: CoroutineContext = EmptyCoroutineContext,
-    dependencies: Collection<Goal<*>> = emptyList(),
+    dependencies: Iterable<Goal<*>> = emptyList(),
     noinline block: suspend () -> T,
 ): Data<T> = Data(typeOf<T>(), meta, context, dependencies, block)

dataforge-data/src/commonMain/kotlin/space/kscience/dataforge/data/DataFilter.kt

@@ -0,0 +1,82 @@
+package space.kscience.dataforge.data
+
+import kotlinx.coroutines.flow.Flow
+import kotlinx.coroutines.flow.filter
+import space.kscience.dataforge.meta.Meta
+import space.kscience.dataforge.misc.DFInternal
+import space.kscience.dataforge.names.Name
+import space.kscience.dataforge.names.NameToken
+import space.kscience.dataforge.names.plus
+import kotlin.reflect.KType
+
+public fun interface DataFilter {
+
+    public fun accepts(name: Name, meta: Meta?, type: KType): Boolean
+
+    public companion object {
+        public val EMPTY: DataFilter = DataFilter { _, _, _ -> true }
+    }
+}
+
+public fun <T> DataSource<T>.filterData(
+    dataFilter: DataFilter,
+): DataSource<T> = object : DataSource<T> {
+    override val dataType: KType get() = this@filterData.dataType
+
+    override fun read(name: Name): Data<T>? =
+        this@filterData.read(name)?.takeIf {
+            dataFilter.accepts(name, it.meta, it.type)
+        }
+}
+
+/**
+ * Stateless filtered [ObservableDataSource]
+ */
+public fun <T> ObservableDataSource<T>.filterData(
+    predicate: DataFilter,
+): ObservableDataSource<T> = object : ObservableDataSource<T> {
+
+    override val updates: Flow<Name>
+        get() = this@filterData.updates.filter {
+            val data = read(it)
+            predicate.accepts(it, data?.meta, data?.type ?: dataType)
+        }
+
+    override val dataType: KType get() = this@filterData.dataType
+
+    override fun read(name: Name): Data<T>? =
+        this@filterData.read(name)?.takeIf { predicate.accepts(name, it.meta, it.type) }
+}
+
+/**
+ * A [DataTree] filtered by branch and some criterion, possibly changing resulting type
+ */
+@DFInternal
+public class FilteredDataTree<T>(
+    public val source: DataTree<T>,
+    public val filter: DataFilter,
+    public val branch: Name,
+    override val dataType: KType = source.dataType,
+) : DataTree<T> {
+
+    override val data: Data<T>?
+        get() = source[branch].takeIf {
+            filter.accepts(Name.EMPTY, it?.meta, it?.type ?: dataType)
+        }
+
+    override val items: Map<NameToken, DataTree<T>>
+        get() = source.branch(branch)?.items
+            ?.mapValues { FilteredDataTree(source, filter, branch + it.key) }
+            ?.filter { !it.value.isEmpty() }
+            ?: emptyMap()
+
+    override val updates: Flow<Name>
+        get() = source.updates.filter {
+            val data = read(it)
+            filter.accepts(it, data?.meta, data?.type ?: dataType)
+        }
+}
+
+public fun <T> DataTree<T>.filterData(
+    predicate: DataFilter,
+): FilteredDataTree<T> = FilteredDataTree(this, predicate, Name.EMPTY)

dataforge-data/src/commonMain/kotlin/space/kscience/dataforge/data/DataRenamer.kt

@@ -0,0 +1,56 @@
+/* 
+ * Copyright 2015 Alexander Nozik.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package space.kscience.dataforge.data
+
+import space.kscience.dataforge.meta.Meta
+import space.kscience.dataforge.meta.get
+import space.kscience.dataforge.meta.string
+import space.kscience.dataforge.misc.DFExperimental
+import space.kscience.dataforge.misc.UnsafeKType
+import space.kscience.dataforge.names.Name
+import space.kscience.dataforge.names.NameToken
+import space.kscience.dataforge.names.plus
+import kotlin.reflect.KType
+
+/**
+ * Interface that define rename rule for [Data]
+ */
+@DFExperimental
+public fun interface DataRenamer {
+    public fun rename(name: Name, meta: Meta, type: KType): Name
+
+    public companion object {
+
+        /**
+         * Prepend name token `key\[tagValue\]` to data name
+         */
+        @OptIn(UnsafeKType::class)
+        public fun groupByMetaValue(
+            key: String,
+            defaultTagValue: String,
+        ): DataRenamer = object : DataRenamer {
+
+            override fun rename(
+                name: Name,
+                meta: Meta,
+                type: KType
+            ): Name {
+                val tagValue: String = meta[key]?.string ?: defaultTagValue
+                return NameToken(key,tagValue).plus(name)
+            }
+        }
+    }
+}

dataforge-data/src/commonMain/kotlin/space/kscience/dataforge/data/DataSet.kt

@@ -1,124 +0,0 @@
-package space.kscience.dataforge.data
-
-import kotlinx.coroutines.*
-import kotlinx.coroutines.flow.Flow
-import kotlinx.coroutines.flow.emptyFlow
-import kotlinx.coroutines.flow.mapNotNull
-import space.kscience.dataforge.data.Data.Companion.TYPE_OF_NOTHING
-import space.kscience.dataforge.meta.Meta
-import space.kscience.dataforge.names.Name
-import space.kscience.dataforge.names.asName
-import space.kscience.dataforge.names.endsWith
-import space.kscience.dataforge.names.parseAsName
-import kotlin.reflect.KType
-
-public interface DataSet<out T : Any> {
-
-    /**
-     * The minimal common ancestor to all data in the node
-     */
-    public val dataType: KType
-
-    /**
-     * Meta-data associated with this node. If no meta is provided, returns [Meta.EMPTY].
-     */
-    public val meta: Meta
-
-    /**
-     * Traverse this [DataSet] returning named data instances. The order is not guaranteed.
-     */
-    public operator fun iterator(): Iterator<NamedData<T>>
-
-    /**
-     * Get data with given name.
-     */
-    public operator fun get(name: Name): Data<T>?
-
-    public companion object {
-        public val META_KEY: Name = "@meta".asName()
-
-        /**
-         * An empty [DataSet] that suits all types
-         */
-        public val EMPTY: DataSet<Nothing> = object : DataSet<Nothing> {
-            override val dataType: KType = TYPE_OF_NOTHING
-            override val meta: Meta get() = Meta.EMPTY
-
-            override fun iterator(): Iterator<NamedData<Nothing>> = emptySequence<NamedData<Nothing>>().iterator()
-
-            override fun get(name: Name): Data<Nothing>? = null
-        }
-    }
-}
-
-public fun <T : Any> DataSet<T>.asSequence(): Sequence<NamedData<T>> = object : Sequence<NamedData<T>> {
-    override fun iterator(): Iterator<NamedData<T>> = this@asSequence.iterator()
-}
-
-/**
- * Return a single [Data] in this [DataSet]. Throw error if it is not single.
- */
-public fun <T : Any> DataSet<T>.single(): NamedData<T> = asSequence().single()
-
-public fun <T : Any> DataSet<T>.asIterable(): Iterable<NamedData<T>> = object : Iterable<NamedData<T>> {
-    override fun iterator(): Iterator<NamedData<T>> = this@asIterable.iterator()
-}
-
-public operator fun <T : Any> DataSet<T>.get(name: String): Data<T>? = get(name.parseAsName())
-
-/**
- * A [DataSet] with propagated updates.
- */
-public interface DataSource<out T : Any> : DataSet<T>, CoroutineScope {
-
-    /**
-     * A flow of updated item names. Updates are propagated in a form of [Flow] of names of updated nodes.
-     * Those can include new data items and replacement of existing ones. The replaced items could update existing data content
-     * and replace it completely, so they should be pulled again.
-     *
-     */
-    public val updates: Flow<Name>
-
-    /**
-     * Stop generating updates from this [DataSource]
-     */
-    public fun close() {
-        coroutineContext[Job]?.cancel()
-    }
-}
-
-public val <T : Any> DataSet<T>.updates: Flow<Name> get() = if (this is DataSource) updates else emptyFlow()
-//
-///**
-// * Flow all data nodes with names starting with [branchName]
-// */
-//public fun <T : Any> DataSet<T>.children(branchName: Name): Sequence<NamedData<T>> =
-//    this@children.asSequence().filter {
-//        it.name.startsWith(branchName)
-//    }
-
-/**
- * Start computation for all goals in data node and return a job for the whole node
- */
-public fun <T : Any> DataSet<T>.startAll(coroutineScope: CoroutineScope): Job = coroutineScope.launch {
-    asIterable().map {
-        it.launch(this@launch)
-    }.joinAll()
-}
-
-public suspend fun <T : Any> DataSet<T>.computeAndJoinAll(): Unit = coroutineScope { startAll(this).join() }
-
-public fun DataSet<*>.toMeta(): Meta = Meta {
-    forEach {
-        if (it.name.endsWith(DataSet.META_KEY)) {
-            set(it.name, it.meta)
-        } else {
-            it.name put {
-                "type" put it.type.toString()
-                "meta" put it.meta
-            }
-        }
-    }
-}
-
-public val <T : Any> DataSet<T>.updatesWithData: Flow<NamedData<T>> get() = updates.mapNotNull { get(it)?.named(it) }

dataforge-data/src/commonMain/kotlin/space/kscience/dataforge/data/DataSetBuilder.kt

@@ -1,165 +0,0 @@
-package space.kscience.dataforge.data
-
-import space.kscience.dataforge.meta.Meta
-import space.kscience.dataforge.meta.MutableMeta
-import space.kscience.dataforge.misc.DFExperimental
-import space.kscience.dataforge.names.Name
-import space.kscience.dataforge.names.isEmpty
-import space.kscience.dataforge.names.plus
-import kotlin.reflect.KType
-
-public interface DataSetBuilder<in T : Any> {
-    public val dataType: KType
-
-    /**
-     * Remove all data items starting with [name]
-     */
-    public fun remove(name: Name)
-
-    public fun data(name: Name, data: Data<T>?)
-
-    /**
-     * Set a current state of given [dataSet] into a branch [name]. Does not propagate updates
-     */
-    public fun node(name: Name, dataSet: DataSet<T>) {
-        //remove previous items
-        if (name != Name.EMPTY) {
-            remove(name)
-        }
-
-        //Set new items
-        dataSet.forEach {
-            data(name + it.name, it.data)
-        }
-    }
-
-    /**
-     * Set meta for the given node
-     */
-    public fun meta(name: Name, meta: Meta)
-
-}
-
-/**
- * Define meta in this [DataSet]
- */
-public fun <T : Any> DataSetBuilder<T>.meta(value: Meta): Unit = meta(Name.EMPTY, value)
-
-/**
- * Define meta in this [DataSet]
- */
-public fun <T : Any> DataSetBuilder<T>.meta(mutableMeta: MutableMeta.() -> Unit): Unit = meta(Meta(mutableMeta))
-
-@PublishedApi
-internal class SubSetBuilder<in T : Any>(
-    private val parent: DataSetBuilder<T>,
-    private val branch: Name,
-) : DataSetBuilder<T> {
-    override val dataType: KType get() = parent.dataType
-
-    override fun remove(name: Name) {
-        parent.remove(branch + name)
-    }
-
-    override fun data(name: Name, data: Data<T>?) {
-        parent.data(branch + name, data)
-    }
-
-    override fun node(name: Name, dataSet: DataSet<T>) {
-        parent.node(branch + name, dataSet)
-    }
-
-    override fun meta(name: Name, meta: Meta) {
-        parent.meta(branch + name, meta)
-    }
-}
-
-public inline fun <T : Any> DataSetBuilder<T>.node(
-    name: Name,
-    crossinline block: DataSetBuilder<T>.() -> Unit,
-) {
-    if (name.isEmpty()) block() else SubSetBuilder(this, name).block()
-}
-
-
-public fun <T : Any> DataSetBuilder<T>.data(name: String, value: Data<T>) {
-    data(Name.parse(name), value)
-}
-
-public fun <T : Any> DataSetBuilder<T>.node(name: String, set: DataSet<T>) {
-    node(Name.parse(name), set)
-}
-
-public inline fun <T : Any> DataSetBuilder<T>.node(
-    name: String,
-    crossinline block: DataSetBuilder<T>.() -> Unit,
-): Unit = node(Name.parse(name), block)
-
-public fun <T : Any> DataSetBuilder<T>.set(value: NamedData<T>) {
-    data(value.name, value.data)
-}
-
-/**
- * Produce lazy [Data] and emit it into the [DataSetBuilder]
- */
-public inline fun <reified T : Any> DataSetBuilder<T>.produce(
-    name: String,
-    meta: Meta = Meta.EMPTY,
-    noinline producer: suspend () -> T,
-) {
-    val data = Data(meta, block = producer)
-    data(name, data)
-}
-
-public inline fun <reified T : Any> DataSetBuilder<T>.produce(
-    name: Name,
-    meta: Meta = Meta.EMPTY,
-    noinline producer: suspend () -> T,
-) {
-    val data = Data(meta, block = producer)
-    data(name, data)
-}
-
-/**
- * Emit a static data with the fixed value
- */
-public inline fun <reified T : Any> DataSetBuilder<T>.static(
-    name: String,
-    data: T,
-    meta: Meta = Meta.EMPTY,
-): Unit = data(name, Data.static(data, meta))
-
-public inline fun <reified T : Any> DataSetBuilder<T>.static(
-    name: Name,
-    data: T,
-    meta: Meta = Meta.EMPTY,
-): Unit = data(name, Data.static(data, meta))
-
-public inline fun <reified T : Any> DataSetBuilder<T>.static(
-    name: String,
-    data: T,
-    mutableMeta: MutableMeta.() -> Unit,
-): Unit = data(Name.parse(name), Data.static(data, Meta(mutableMeta)))
-
-/**
- * Update data with given node data and meta with node meta.
- */
-@DFExperimental
-public fun <T : Any> DataSetBuilder<T>.populateFrom(tree: DataSet<T>): Unit {
-    tree.forEach {
-        //TODO check if the place is occupied
-        data(it.name, it.data)
-    }
-}
-
-//public fun <T : Any> DataSetBuilder<T>.populateFrom(flow: Flow<NamedData<T>>) {
-//    flow.collect {
-//        data(it.name, it.data)
-//    }
-//}
-
-public fun <T : Any> DataSetBuilder<T>.populateFrom(sequence: Sequence<NamedData<T>>) {
-    sequence.forEach {
-        data(it.name, it.data)
-    }
-}

dataforge-data/src/commonMain/kotlin/space/kscience/dataforge/data/DataSink.kt

@@ -0,0 +1,133 @@
+package space.kscience.dataforge.data
+
+import kotlinx.coroutines.flow.Flow
+import kotlinx.coroutines.flow.MutableSharedFlow
+import kotlinx.coroutines.flow.mapNotNull
+import space.kscience.dataforge.misc.UnsafeKType
+import space.kscience.dataforge.names.*
+import kotlin.reflect.KType
+import kotlin.reflect.typeOf
+
+/**
+ * A marker scope for data builders
+ */
+public interface DataBuilderScope<in T> {
+    public companion object : DataBuilderScope<Nothing>
+}
+
+@Suppress("UNCHECKED_CAST")
+public fun <T> DataBuilderScope(): DataBuilderScope<T> = DataBuilderScope as DataBuilderScope<T>
+
+/**
+ * Asynchronous data sink
+ */
+public fun interface DataSink<in T> : DataBuilderScope<T> {
+    /**
+     * Put data and notify listeners if needed
+     */
+    public suspend fun write(name: Name, data: Data<T>?)
+}
+
+
+/**
+ * A mutable version of [DataTree]
+ */
+public interface MutableDataTree<T> : DataTree<T>, DataSink<T> {
+    override val items: Map<NameToken, MutableDataTree<T>>
+//
+//    public fun getOrCreateItem(token: NameToken): MutableDataTree<T>
+//
+//    public suspend fun put(token: NameToken, data: Data<T>?)
+//
+//    override suspend fun put(name: Name, data: Data<T>?): Unit {
+//        when (name.length) {
+//            0 -> this.data = data
+//            1 -> put(name.first(), data)
+//            else -> getOrCreateItem(name.first()).put(name.cutFirst(), data)
+//        }
+//    }
+}
+
+/**
+ * Provide a mutable subtree if it exists
+ */
+public tailrec fun <T> MutableDataTree<T>.branch(name: Name): MutableDataTree<T>? =
+    when (name.length) {
+        0 -> this
+        1 -> items[name.first()]
+        else -> items[name.first()]?.branch(name.cutFirst())
+    }
+
+private class MutableDataTreeRoot<T>(
+    override val dataType: KType,
+) : MutableDataTree<T> {
+
+    override val items = HashMap<NameToken, MutableDataTree<T>>()
+    override val updates = MutableSharedFlow<Name>()
+
+    inner class MutableDataTreeBranch(val branchName: Name) : MutableDataTree<T> {
+
+        override var data: Data<T>? = null
+            private set
+
+        override val items = HashMap<NameToken, MutableDataTree<T>>()
+
+        override val updates: Flow<Name> = this@MutableDataTreeRoot.updates.mapNotNull { update ->
+            update.removeFirstOrNull(branchName)
+        }
+        override val dataType: KType get() = this@MutableDataTreeRoot.dataType
+
+        override suspend fun write(
+            name: Name,
+            data: Data<T>?
+        ) {
+            when (name.length) {
+                0 -> {
+                    this.data = data
+                    this@MutableDataTreeRoot.updates.emit(branchName)
+                }
+
+                else -> {
+                    val token = name.first()
+                    items.getOrPut(token) { MutableDataTreeBranch(branchName + token) }.write(name.cutFirst(), data)
+                }
+            }
+        }
+    }
+    override var data: Data<T>? = null
+        private set
+
+    override suspend fun write(
+        name: Name,
+        data: Data<T>?
+    ) {
+        when (name.length) {
+            0 -> {
+                this.data = data
+                this@MutableDataTreeRoot.updates.emit(Name.EMPTY)
+            }
+
+            else -> {
+                val token = name.first()
+                items.getOrPut(token) { MutableDataTreeBranch(token.asName()) }.write(name.cutFirst(), data)
+            }
+        }
+    }
+
+}
+
+/**
+ * Create a new [MutableDataTree]
+ */
+@UnsafeKType
+public fun <T> MutableDataTree(
+    type: KType,
+): MutableDataTree<T> = MutableDataTreeRoot<T>(type)
+
+/**
+ * Create and initialize an observable mutable data tree.
+ */
+@OptIn(UnsafeKType::class)
+public inline fun <reified T> MutableDataTree(
+    generator: MutableDataTree<T>.() -> Unit = {},
+): MutableDataTree<T> = MutableDataTree<T>(typeOf<T>()).apply { generator() }

dataforge-data/src/commonMain/kotlin/space/kscience/dataforge/data/DataSource.kt

@@ -0,0 +1,139 @@
+package space.kscience.dataforge.data
+
+import kotlinx.coroutines.flow.*
+import space.kscience.dataforge.meta.Meta
+import space.kscience.dataforge.names.*
+import kotlin.contracts.contract
+import kotlin.reflect.KType
+import kotlin.reflect.typeOf
+
+/**
+ * A generic data provider
+ */
+public interface DataSource<out T> {
+
+    /**
+     * The minimal common ancestor to all data in the node
+     */
+    public val dataType: KType
+
+    /**
+     * Get data with given name. Or null if it is not present
+     */
+    public fun read(name: Name): Data<T>?
+}
+
+/**
+ * A data provider with possible dynamic updates
+ */
+public interface ObservableDataSource<out T> : DataSource<T> {
+
+    /**
+     * Names of updated elements.
+     * Data updates with the same names could be glued together.
+     *
+     * Updates are considered critical.
+     * The producer will suspend unless all updates are consumed.
+     */
+    public val updates: Flow<Name>
+}
+
+public suspend fun <T> ObservableDataSource<T>.awaitData(name: Name): Data<T> =
+    read(name) ?: updates.filter { it == name }.mapNotNull { read(name) }.first()
+
+public suspend fun <T> ObservableDataSource<T>.awaitData(name: String): Data<T> =
+    awaitData(name.parseAsName())
+
+/**
+ * A tree like structure for data holding
+ */
+public interface DataTree<out T> : ObservableDataSource<T> {
+
+    public val data: Data<T>?
+    public val items: Map<NameToken, DataTree<T>>
+
+    override fun read(name: Name): Data<T>? = when (name.length) {
+        0 -> data
+        else -> items[name.first()]?.read(name.cutFirst())
+    }
+
+    /**
+     * Flow updates made to the data
+     */
+    override val updates: Flow<Name>
+
+    public companion object {
+        private object EmptyDataTree : DataTree<Nothing> {
+            override val data: Data<Nothing>? = null
+            override val items: Map<NameToken, EmptyDataTree> = emptyMap()
+            override val dataType: KType = typeOf<Unit>()
+
+            override fun read(name: Name): Data<Nothing>? = null
+            override val updates: Flow<Name> get() = emptyFlow()
+        }
+
+        public val EMPTY: DataTree<Nothing> = EmptyDataTree
+    }
+}
+
+/**
+ * An alias for easier access to tree values
+ */
+public operator fun <T> DataTree<T>.get(name: Name): Data<T>? = read(name)
+
+public operator fun <T> DataTree<T>.get(name: String): Data<T>? = read(name.parseAsName())
+
+/**
+ * Return a sequence of all data items in this tree.
+ * This method does not take updates into account.
+ */
+public fun <T> DataTree<T>.asSequence(
+    namePrefix: Name = Name.EMPTY,
+): Sequence<NamedData<T>> = sequence {
+    data?.let { yield(it.named(namePrefix)) }
+    items.forEach { (token, tree) ->
+        yieldAll(tree.asSequence(namePrefix + token))
+    }
+}
+
+/**
+ * Walk the data tree depth-first.
+ *
+ * @return a [Sequence] of pairs [Name]-[DataTree] for all nodes including the root one.
+ */
+public fun <T> DataTree<T>.walk(
+    namePrefix: Name = Name.EMPTY,
+): Sequence<Pair<Name, DataTree<T>>> = sequence {
+    yield(namePrefix to this@walk)
+    items.forEach { (token, tree) ->
+        yieldAll(tree.walk(namePrefix + token))
+    }
+}
+
+public val DataTree<*>.meta: Meta? get() = data?.meta
+
+/**
+ * Provide subtree if it exists
+ */
+public tailrec fun <T> DataTree<T>.branch(name: Name): DataTree<T>? =
+    when (name.length) {
+        0 -> this
+        1 -> items[name.first()]
+        else -> items[name.first()]?.branch(name.cutFirst())
+    }
+
+public fun <T> DataTree<T>.branch(name: String): DataTree<T>? =
+    branch(name.parseAsName())
+
+public fun DataTree<*>.isEmpty(): Boolean = data == null && items.isEmpty()
+
+/**
+ * Check if the [DataTree] is observable
+ */
+public fun <T> DataSource<T>.isObservable(): Boolean {
+    contract {
+        returns(true) implies (this@isObservable is ObservableDataSource<T>)
+    }
+    return this is ObservableDataSource<T>
+}
+

dataforge-data/src/commonMain/kotlin/space/kscience/dataforge/data/DataTree.kt

@@ -1,119 +0,0 @@
-package space.kscience.dataforge.data
-
-import space.kscience.dataforge.meta.Meta
-import space.kscience.dataforge.misc.DFInternal
-import space.kscience.dataforge.misc.DfId
-import space.kscience.dataforge.names.*
-import kotlin.collections.component1
-import kotlin.collections.component2
-import kotlin.reflect.KType
-import kotlin.reflect.typeOf
-
-public sealed class DataTreeItem<out T : Any> {
-
-    public abstract val meta: Meta
-
-    public class Node<out T : Any>(public val tree: DataTree<T>) : DataTreeItem<T>() {
-        override val meta: Meta get() = tree.meta
-    }
-
-    public class Leaf<out T : Any>(public val data: Data<T>) : DataTreeItem<T>() {
-        override val meta: Meta get() = data.meta
-    }
-}
-
-public val <T : Any> DataTreeItem<T>.type: KType
-    get() = when (this) {
-        is DataTreeItem.Node -> tree.dataType
-        is DataTreeItem.Leaf -> data.type
-    }
-
-/**
- * A tree-like [DataSet] grouped into the node. All data inside the node must inherit its type
- */
-@DfId(DataTree.TYPE)
-public interface DataTree<out T : Any> : DataSet<T> {
-
-    /**
-     * Top-level children items of this [DataTree]
-     */
-    public val items: Map<NameToken, DataTreeItem<T>>
-
-    override val meta: Meta get() = items[META_ITEM_NAME_TOKEN]?.meta ?: Meta.EMPTY
-
-    override fun iterator(): Iterator<NamedData<T>> = iterator {
-        items.forEach { (token, childItem: DataTreeItem<T>) ->
-            if (!token.body.startsWith("@")) {
-                when (childItem) {
-                    is DataTreeItem.Leaf -> yield(childItem.data.named(token.asName()))
-                    is DataTreeItem.Node -> yieldAll(childItem.tree.asSequence().map { it.named(token + it.name) })
-                }
-            }
-        }
-    }
-
-    override fun get(name: Name): Data<T>? = when (name.length) {
-        0 -> null
-        1 -> items[name.firstOrNull()!!].data
-        else -> items[name.firstOrNull()!!].tree?.get(name.cutFirst())
-    }
-
-    public companion object {
-        public const val TYPE: String = "dataTree"
-
-        /**
-         * A name token used to designate tree node meta
-         */
-        public val META_ITEM_NAME_TOKEN: NameToken = NameToken("@meta")
-
-        @DFInternal
-        public fun <T : Any> emptyWithType(type: KType, meta: Meta = Meta.EMPTY): DataTree<T> = object : DataTree<T> {
-            override val items: Map<NameToken, DataTreeItem<T>> get() = emptyMap()
-            override val dataType: KType get() = type
-            override val meta: Meta get() = meta
-        }
-
-        @OptIn(DFInternal::class)
-        public inline fun <reified T : Any> empty(meta: Meta = Meta.EMPTY): DataTree<T> =
-            emptyWithType<T>(typeOf<T>(), meta)
-    }
-}
-
-public fun <T : Any> DataTree<T>.listChildren(prefix: Name): List<Name> =
-    getItem(prefix).tree?.items?.keys?.map { prefix + it } ?: emptyList()
-
-/**
- * Get a [DataTreeItem] with given [name] or null if the item does not exist
- */
-public tailrec fun <T : Any> DataTree<T>.getItem(name: Name): DataTreeItem<T>? = when (name.length) {
-    0 -> DataTreeItem.Node(this)
-    1 -> items[name.firstOrNull()]
-    else -> items[name.firstOrNull()!!].tree?.getItem(name.cutFirst())
-}
-
-public val <T : Any> DataTreeItem<T>?.tree: DataTree<T>? get() = (this as? DataTreeItem.Node<T>)?.tree
-public val <T : Any> DataTreeItem<T>?.data: Data<T>? get() = (this as? DataTreeItem.Leaf<T>)?.data
-
-/**
- * A [Sequence] of all children including nodes
- */
-public fun <T : Any> DataTree<T>.traverseItems(): Sequence<Pair<Name, DataTreeItem<T>>> = sequence {
-    items.forEach { (head, item) ->
-        yield(head.asName() to item)
-        if (item is DataTreeItem.Node) {
-            val subSequence = item.tree.traverseItems()
-                .map { (name, data) -> (head.asName() + name) to data }
-            yieldAll(subSequence)
-        }
-    }
-}
-
-/**
- * Get a branch of this [DataTree] with a given [branchName].
- * The difference from similar method for [DataSet] is that internal logic is more simple and the return value is a [DataTree]
- */
-@OptIn(DFInternal::class)
-public fun <T : Any> DataTree<T>.branch(branchName: Name): DataTree<T> =
-    getItem(branchName)?.tree ?: DataTree.emptyWithType(dataType)
-
-public fun <T : Any> DataTree<T>.branch(branchName: String): DataTree<T> = branch(branchName.parseAsName())

dataforge-data/src/commonMain/kotlin/space/kscience/dataforge/data/DataTreeBuilder.kt

@@ -1,127 +0,0 @@
-package space.kscience.dataforge.data
-
-import kotlinx.coroutines.CoroutineScope
-import kotlinx.coroutines.Job
-import kotlinx.coroutines.flow.MutableSharedFlow
-import kotlinx.coroutines.launch
-import space.kscience.dataforge.meta.Meta
-import space.kscience.dataforge.misc.DFInternal
-import space.kscience.dataforge.misc.ThreadSafe
-import space.kscience.dataforge.names.*
-import kotlin.collections.set
-import kotlin.coroutines.CoroutineContext
-import kotlin.coroutines.coroutineContext
-import kotlin.reflect.KType
-import kotlin.reflect.typeOf
-
-public interface DataSourceBuilder<T : Any> : DataSetBuilder<T>, DataSource<T> {
-    override val updates: MutableSharedFlow<Name>
-}
-
-/**
- * A mutable [DataTree] that propagates updates
- */
-public class DataTreeBuilder<T : Any>(
-    override val dataType: KType,
-    coroutineContext: CoroutineContext,
-) : DataTree<T>, DataSourceBuilder<T> {
-
-    override val coroutineContext: CoroutineContext =
-        coroutineContext + Job(coroutineContext[Job]) + GoalExecutionRestriction()
-
-    private val treeItems = HashMap<NameToken, DataTreeItem<T>>()
-
-    override val items: Map<NameToken, DataTreeItem<T>>
-        get() = treeItems.filter { !it.key.body.startsWith("@") }
-
-    override val updates: MutableSharedFlow<Name> = MutableSharedFlow<Name>()
-
-    @ThreadSafe
-    private fun remove(token: NameToken) {
-        if (treeItems.remove(token) != null) {
-            launch {
-                updates.emit(token.asName())
-            }
-        }
-    }
-
-    override fun remove(name: Name) {
-        if (name.isEmpty()) error("Can't remove the root node")
-        (getItem(name.cutLast()).tree as? DataTreeBuilder)?.remove(name.lastOrNull()!!)
-    }
-
-    @ThreadSafe
-    private fun set(token: NameToken, data: Data<T>) {
-        treeItems[token] = DataTreeItem.Leaf(data)
-    }
-
-    @ThreadSafe
-    private fun set(token: NameToken, node: DataTree<T>) {
-        treeItems[token] = DataTreeItem.Node(node)
-    }
-
-    private fun getOrCreateNode(token: NameToken): DataTreeBuilder<T> =
-        (treeItems[token] as? DataTreeItem.Node<T>)?.tree as? DataTreeBuilder<T>
-            ?: DataTreeBuilder<T>(dataType, coroutineContext).also { set(token, it) }
-
-    private fun getOrCreateNode(name: Name): DataTreeBuilder<T> = when (name.length) {
-        0 -> this
-        1 -> getOrCreateNode(name.firstOrNull()!!)
-        else -> getOrCreateNode(name.firstOrNull()!!).getOrCreateNode(name.cutFirst())
-    }
-
-    override fun data(name: Name, data: Data<T>?) {
-        if (data == null) {
-            remove(name)
-        } else {
-            when (name.length) {
-                0 -> error("Can't add data with empty name")
-                1 -> set(name.firstOrNull()!!, data)
-                2 -> getOrCreateNode(name.cutLast()).set(name.lastOrNull()!!, data)
-            }
-        }
-        launch {
-            updates.emit(name)
-        }
-    }
-
-    override fun meta(name: Name, meta: Meta) {
-        val item = getItem(name)
-        if (item is DataTreeItem.Leaf) error("TODO: Can't change meta of existing leaf item.")
-        data(name + DataTree.META_ITEM_NAME_TOKEN, Data.empty(meta))
-    }
-}
-
-/**
- * Create a dynamic [DataSource]. Initial data is placed synchronously.
- */
-@DFInternal
-@Suppress("FunctionName")
-public fun <T : Any> DataSource(
-    type: KType,
-    parent: CoroutineScope,
-    block: DataSourceBuilder<T>.() -> Unit,
-): DataTreeBuilder<T> = DataTreeBuilder<T>(type, parent.coroutineContext).apply(block)
-
-@Suppress("OPT_IN_USAGE", "FunctionName")
-public inline fun <reified T : Any> DataSource(
-    parent: CoroutineScope,
-    crossinline block: DataSourceBuilder<T>.() -> Unit,
-): DataTreeBuilder<T> = DataSource(typeOf<T>(), parent) { block() }
-
-@Suppress("FunctionName")
-public suspend inline fun <reified T : Any> DataSource(
-    crossinline block: DataSourceBuilder<T>.() -> Unit = {},
-): DataTreeBuilder<T> = DataTreeBuilder<T>(typeOf<T>(), coroutineContext).apply { block() }
-
-public inline fun <reified T : Any> DataSourceBuilder<T>.emit(
-    name: Name,
-    parent: CoroutineScope,
-    noinline block: DataSourceBuilder<T>.() -> Unit,
-): Unit = node(name, DataSource(parent, block))
-
-public inline fun <reified T : Any> DataSourceBuilder<T>.emit(
-    name: String,
-    parent: CoroutineScope,
-    noinline block: DataSourceBuilder<T>.() -> Unit,
-): Unit = node(Name.parse(name), DataSource(parent, block))

dataforge-data/src/commonMain/kotlin/space/kscience/dataforge/data/Goal.kt

@@ -9,7 +9,7 @@ import kotlin.coroutines.EmptyCoroutineContext
  * Lazy computation result with its dependencies to allowing to stat computing dependencies ahead of time
  */
 public interface Goal<out T> {
-    public val dependencies: Collection<Goal<*>>
+    public val dependencies: Iterable<Goal<*>>
 
     /**
      * Returns current running coroutine if the goal is started. Null if the computation is not started.
@@ -32,7 +32,7 @@ public interface Goal<out T> {
     public companion object
 }
 
-public fun Goal<*>.launch(coroutineScope: CoroutineScope): Job = async(coroutineScope)
+public fun Goal<*>.launchIn(coroutineScope: CoroutineScope): Job = async(coroutineScope)
 
 public suspend fun <T> Goal<T>.await(): T = coroutineScope { async(this).await() }
 
@@ -54,7 +54,7 @@ public open class StaticGoal<T>(public val value: T) : Goal<T> {
  */
 public open class LazyGoal<T>(
     private val coroutineContext: CoroutineContext = EmptyCoroutineContext,
-    override val dependencies: Collection<Goal<*>> = emptyList(),
+    override val dependencies: Iterable<Goal<*>> = emptyList(),
     public val block: suspend () -> T,
 ) : Goal<T> {
 
@@ -64,11 +64,14 @@ public open class LazyGoal<T>(
     /**
      * Get ongoing computation or start a new one.
      * Does not guarantee thread safety. In case of multi-thread access, could create orphan computations.
-     * If [GoalExecutionRestriction] is present in the [coroutineScope] context, the call could produce a error a warning
+     * If [GoalExecutionRestriction] is present in the [coroutineScope] context, the call could produce an error or a warning
      * depending on the settings.
+     *
+     * If [Goal] is already started on a different scope, it is not restarted.
      */
     @OptIn(DFExperimental::class)
-    override fun async(coroutineScope: CoroutineScope): Deferred<T> {
+    override fun async(coroutineScope: CoroutineScope): Deferred<T> = deferred ?: run {
+
         val log = coroutineScope.coroutineContext[GoalLogger]
         // Check if context restricts goal computation
         coroutineScope.coroutineContext[GoalExecutionRestriction]?.let { restriction ->
@@ -82,16 +85,17 @@ public open class LazyGoal<T>(
         }
 
         log?.emit { "Starting dependencies computation for ${this@LazyGoal}" }
-        val startedDependencies = this.dependencies.map { goal ->
+        val startedDependencies = dependencies.map { goal ->
-            goal.run { async(coroutineScope) }
+            goal.async(coroutineScope)
         }
-        return deferred ?: coroutineScope.async(
+
+        coroutineScope.async(
             coroutineContext
                     + CoroutineMonitor()
                     + Dependencies(startedDependencies)
                     + GoalExecutionRestriction(GoalExecutionRestrictionPolicy.NONE) // Remove restrictions on goal execution
         ) {
-            //cancel execution if error encountered in one of dependencies
+            //cancel execution if error encountered in one of the dependencies
             startedDependencies.forEach { deferred ->
                 deferred.invokeOnCompletion { error ->
                     if (error != null) this.cancel(CancellationException("Dependency $deferred failed with error: ${error.message}"))

dataforge-data/src/commonMain/kotlin/space/kscience/dataforge/data/GroupRule.kt

@@ -1,78 +0,0 @@
-/* 
- * Copyright 2015 Alexander Nozik.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *      http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package space.kscience.dataforge.data
-
-import kotlinx.coroutines.launch
-import space.kscience.dataforge.meta.get
-import space.kscience.dataforge.meta.string
-import space.kscience.dataforge.misc.DFInternal
-
-public interface GroupRule {
-    public fun <T : Any> gather(set: DataSet<T>): Map<String, DataSet<T>>
-
-    public companion object {
-        /**
-         * Create grouping rule that creates groups for different values of value
-         * field with name [key]
-         *
-         * @param key
-         * @param defaultTagValue
-         * @return
-         */
-        @OptIn(DFInternal::class)
-        public fun byMetaValue(
-            key: String,
-            defaultTagValue: String,
-        ): GroupRule = object : GroupRule {
-
-            override fun <T : Any> gather(
-                set: DataSet<T>,
-            ): Map<String, DataSet<T>> {
-                val map = HashMap<String, DataSet<T>>()
-
-                if (set is DataSource) {
-                    set.forEach { data ->
-                        val tagValue: String = data.meta[key]?.string ?: defaultTagValue
-                        (map.getOrPut(tagValue) { DataTreeBuilder(set.dataType, set.coroutineContext) } as DataTreeBuilder<T>)
-                            .data(data.name, data.data)
-
-                        set.launch {
-                            set.updates.collect { name ->
-                                val dataUpdate = set[name]
-
-                                val updateTagValue = dataUpdate?.meta?.get(key)?.string ?: defaultTagValue
-                                map.getOrPut(updateTagValue) {
-                                    DataSource(set.dataType, this) {
-                                        data(name, dataUpdate)
-                                    }
-                                }
-                            }
-                        }
-                    }
-                } else {
-                    set.forEach { data ->
-                        val tagValue: String = data.meta[key]?.string ?: defaultTagValue
-                        (map.getOrPut(tagValue) { StaticDataTree(set.dataType) } as StaticDataTree<T>)
-                            .data(data.name, data.data)
-                    }
-                }
-
-
-                return map
-            }
-        }
-    }
-}

dataforge-data/src/commonMain/kotlin/space/kscience/dataforge/data/MetaMaskData.kt

@@ -0,0 +1,23 @@
+package space.kscience.dataforge.data
+
+import space.kscience.dataforge.meta.Meta
+import space.kscience.dataforge.meta.MutableMeta
+import space.kscience.dataforge.meta.copy
+
+
+private class MetaMaskData<T>(val origin: Data<T>, override val meta: Meta) : Data<T> by origin
+
+/**
+ * A data with overridden meta. It reflects original data computed state.
+ */
+public fun <T> Data<T>.withMeta(newMeta: Meta): Data<T> = if (this is MetaMaskData) {
+    MetaMaskData(origin, newMeta)
+} else {
+    MetaMaskData(this, newMeta)
+}
+
+/**
+ * Create a new [Data] with the same computation, but different meta. The meta is created by applying [block] to
+ * the existing data meta.
+ */
+public inline fun <T> Data<T>.withMeta(block: MutableMeta.() -> Unit): Data<T> = withMeta(meta.copy(block))

dataforge-data/src/commonMain/kotlin/space/kscience/dataforge/data/NamedData.kt

@@ -4,17 +4,15 @@ import space.kscience.dataforge.meta.isEmpty
 import space.kscience.dataforge.misc.Named
 import space.kscience.dataforge.names.Name
 
-public interface NamedData<out T : Any> : Named, Data<T> {
+/**
-    override val name: Name
+ * A data coupled to a name.
-    public val data: Data<T>
+ */
-}
+public interface NamedData<out T> : Data<T>, Named
 
-public operator fun NamedData<*>.component1(): Name = name
-public operator fun <T: Any> NamedData<T>.component2(): Data<T> = data
 
-private class NamedDataImpl<out T : Any>(
+private class NamedDataImpl<T>(
     override val name: Name,
-    override val data: Data<T>,
+    val data: Data<T>,
 ) : Data<T> by data, NamedData<T> {
     override fun toString(): String = buildString {
         append("NamedData(name=\"$name\"")
@@ -28,8 +26,10 @@ private class NamedDataImpl<out T : Any>(
     }
 }
 
-public fun <T : Any> Data<T>.named(name: Name): NamedData<T> = if (this is NamedData) {
+public fun <T> Data<T>.named(name: Name): NamedData<T> = if (this is NamedData) {
-    NamedDataImpl(name, this.data)
+    NamedDataImpl(name, this)
 } else {
     NamedDataImpl(name, this)
-}
+}
+
+public fun <T> NamedData(name: Name, data: Data<T>): NamedData<T> = data.named(name)

dataforge-data/src/commonMain/kotlin/space/kscience/dataforge/data/StaticDataBuilder.kt

@@ -0,0 +1,63 @@
+package space.kscience.dataforge.data
+
+import space.kscience.dataforge.meta.Meta
+import space.kscience.dataforge.meta.MutableMeta
+import space.kscience.dataforge.misc.UnsafeKType
+import space.kscience.dataforge.names.Name
+import space.kscience.dataforge.names.parseAsName
+import space.kscience.dataforge.names.plus
+import kotlin.reflect.KType
+import kotlin.reflect.typeOf
+
+
+public fun interface StaticDataBuilder<T> : DataBuilderScope<T> {
+    public fun data(name: Name, data: Data<T>)
+}
+
+private class DataMapBuilder<T> : StaticDataBuilder<T> {
+    val map = mutableMapOf<Name, Data<T>>()
+
+    override fun data(name: Name, data: Data<T>) {
+        if (map.containsKey(name)) {
+            error("Duplicate key '$name'")
+        } else {
+            map.put(name, data)
+        }
+    }
+}
+
+public fun <T> StaticDataBuilder<T>.data(name: String, data: Data<T>) {
+    data(name.parseAsName(), data)
+}
+
+public inline fun <T, reified T1 : T> StaticDataBuilder<T>.value(
+    name: String,
+    value: T1,
+    metaBuilder: MutableMeta.() -> Unit = {}
+) {
+    data(name, Data(value, Meta(metaBuilder)))
+}
+
+public fun <T> StaticDataBuilder<T>.node(prefix: Name, block: StaticDataBuilder<T>.() -> Unit) {
+    val map = DataMapBuilder<T>().apply(block).map
+    map.forEach { (name, data) ->
+        data(prefix + name, data)
+    }
+}
+
+public fun <T> StaticDataBuilder<T>.node(prefix: String, block: StaticDataBuilder<T>.() -> Unit) =
+    node(prefix.parseAsName(), block)
+
+public fun <T> StaticDataBuilder<T>.node(prefix: String, tree: DataTree<T>) {
+    tree.forEach { data ->
+        data(prefix.parseAsName() + data.name, data)
+    }
+}
+
+@UnsafeKType
+public fun <T> DataTree.Companion.static(type: KType, block: StaticDataBuilder<T>.() -> Unit): DataTree<T> =
+    DataMapBuilder<T>().apply(block).map.asTree(type)
+
+@OptIn(UnsafeKType::class)
+public inline fun <reified T> DataTree.Companion.static(noinline block: StaticDataBuilder<T>.() -> Unit): DataTree<T> =
+    static(typeOf<T>(), block)

dataforge-data/src/commonMain/kotlin/space/kscience/dataforge/data/StaticDataTree.kt

@@ -1,82 +0,0 @@
-package space.kscience.dataforge.data
-
-import space.kscience.dataforge.meta.Meta
-import space.kscience.dataforge.misc.DFExperimental
-import space.kscience.dataforge.names.*
-import kotlin.reflect.KType
-import kotlin.reflect.typeOf
-
-@PublishedApi
-internal class StaticDataTree<T : Any>(
-    override val dataType: KType,
-) : DataSetBuilder<T>, DataTree<T> {
-
-    private val _items: MutableMap<NameToken, DataTreeItem<T>> = HashMap()
-
-    override val items: Map<NameToken, DataTreeItem<T>>
-        get() = _items.filter { !it.key.body.startsWith("@") }
-
-    override fun remove(name: Name) {
-        when (name.length) {
-            0 -> error("Can't remove root tree node")
-            1 -> _items.remove(name.firstOrNull()!!)
-            else -> (_items[name.firstOrNull()!!].tree as? StaticDataTree<T>)?.remove(name.cutFirst())
-        }
-    }
-
-    private fun getOrCreateNode(name: Name): StaticDataTree<T> = when (name.length) {
-        0 -> this
-        1 -> {
-            val itemName = name.firstOrNull()!!
-            (_items[itemName].tree as? StaticDataTree<T>) ?: StaticDataTree<T>(dataType).also {
-                _items[itemName] = DataTreeItem.Node(it)
-            }
-        }
-        else -> getOrCreateNode(name.cutLast()).getOrCreateNode(name.lastOrNull()!!.asName())
-    }
-
-    private fun set(name: Name, item: DataTreeItem<T>?) {
-        if (name.isEmpty()) error("Can't set top level tree node")
-        if (item == null) {
-            remove(name)
-        } else {
-            getOrCreateNode(name.cutLast())._items[name.lastOrNull()!!] = item
-        }
-    }
-
-    override fun data(name: Name, data: Data<T>?) {
-        set(name, data?.let { DataTreeItem.Leaf(it) })
-    }
-
-    override fun node(name: Name, dataSet: DataSet<T>) {
-        if (dataSet is StaticDataTree) {
-            set(name, DataTreeItem.Node(dataSet))
-        } else {
-            dataSet.forEach {
-                data(name + it.name, it.data)
-            }
-        }
-    }
-
-    override fun meta(name: Name, meta: Meta) {
-        val item = getItem(name)
-        if (item is DataTreeItem.Leaf) TODO("Can't change meta of existing leaf item.")
-        data(name + DataTree.META_ITEM_NAME_TOKEN, Data.empty(meta))
-    }
-}
-
-@Suppress("FunctionName")
-public inline fun <T : Any> DataTree(
-    dataType: KType,
-    block: DataSetBuilder<T>.() -> Unit,
-): DataTree<T> = StaticDataTree<T>(dataType).apply { block() }
-
-@Suppress("FunctionName")
-public inline fun <reified T : Any> DataTree(
-    noinline block: DataSetBuilder<T>.() -> Unit,
-): DataTree<T> = DataTree(typeOf<T>(), block)
-
-@OptIn(DFExperimental::class)
-public fun <T : Any> DataSet<T>.seal(): DataTree<T> = DataTree(dataType) {
-    populateFrom(this@seal)
-}

dataforge-data/src/commonMain/kotlin/space/kscience/dataforge/data/dataBuilders.kt

@@ -0,0 +1,113 @@
+package space.kscience.dataforge.data
+
+import space.kscience.dataforge.meta.Meta
+import space.kscience.dataforge.meta.MutableMeta
+import space.kscience.dataforge.names.Name
+import space.kscience.dataforge.names.asName
+import space.kscience.dataforge.names.isEmpty
+import space.kscience.dataforge.names.plus
+
+
+public suspend fun <T> DataSink<T>.write(value: NamedData<T>) {
+    write(value.name, value)
+}
+
+public inline fun <T> DataSink<T>.writeAll(
+    prefix: Name,
+    block: DataSink<T>.() -> Unit,
+) {
+    if (prefix.isEmpty()) {
+        apply(block)
+    } else {
+        val proxyDataSink = DataSink<T> { name, data -> this@writeAll.write(prefix + name, data) }
+
+        proxyDataSink.apply(block)
+    }
+}
+
+
+public inline fun <T> DataSink<T>.writeAll(
+    prefix: String,
+    block: DataSink<T>.() -> Unit,
+): Unit = writeAll(prefix.asName(), block)
+
+
+public suspend fun <T> DataSink<T>.write(name: String, value: Data<T>) {
+    write(Name.parse(name), value)
+}
+
+public suspend fun <T> DataSink<T>.writeAll(name: Name, tree: DataTree<T>) {
+    writeAll(name) { writeAll(tree.asSequence()) }
+}
+
+
+public suspend fun <T> DataSink<T>.writeAll(name: String, tree: DataTree<T>) {
+    writeAll(Name.parse(name)) { writeAll(tree.asSequence()) }
+}
+
+/**
+ * Produce lazy [Data] and emit it into the [MutableDataTree]
+ */
+public suspend inline fun <reified T> DataSink<T>.writeValue(
+    name: String,
+    meta: Meta = Meta.EMPTY,
+    noinline producer: suspend () -> T,
+) {
+    val data = Data(meta, block = producer)
+    write(name, data)
+}
+
+public suspend inline fun <reified T> DataSink<T>.writeValue(
+    name: Name,
+    meta: Meta = Meta.EMPTY,
+    noinline producer: suspend () -> T,
+) {
+    val data = Data(meta, block = producer)
+    write(name, data)
+}
+
+/**
+ * Emit static data with the fixed value
+ */
+public suspend inline fun <reified T> DataSink<T>.writeValue(
+    name: Name,
+    value: T,
+    meta: Meta = Meta.EMPTY,
+): Unit = write(name, Data.wrapValue(value, meta))
+
+public suspend inline fun <reified T> DataSink<T>.writeValue(
+    name: String,
+    value: T,
+    meta: Meta = Meta.EMPTY,
+): Unit = write(name, Data.wrapValue(value, meta))
+
+public suspend inline fun <reified T> DataSink<T>.writeValue(
+    name: String,
+    value: T,
+    metaBuilder: MutableMeta.() -> Unit,
+): Unit = write(Name.parse(name), Data.wrapValue(value, Meta(metaBuilder)))
+
+public suspend fun <T> DataSink<T>.writeAll(sequence: Sequence<NamedData<T>>) {
+    sequence.forEach {
+        write(it)
+    }
+}
+
+public suspend fun <T> DataSink<T>.writeAll(map: Map<Name, Data<T>?>) {
+    map.forEach { (name, data) ->
+        write(name, data)
+    }
+}
+
+/**
+ * Copy all data from [this] and mirror changes if they appear. Suspends indefinitely.
+ */
+public suspend fun <T : Any> MutableDataTree<T>.writeAllAndWatch(
+    source: DataTree<T>,
+    prefix: Name = Name.EMPTY,
+) {
+    writeAll(prefix, source)
+    source.updates.collect {
+        write(prefix + it, source.read(it))
+    }
+}

dataforge-data/src/commonMain/kotlin/space/kscience/dataforge/data/dataFilter.kt

@@ -1,105 +0,0 @@
-package space.kscience.dataforge.data
-
-import kotlinx.coroutines.flow.Flow
-import kotlinx.coroutines.flow.filter
-import kotlinx.coroutines.flow.map
-import kotlinx.coroutines.flow.mapNotNull
-import space.kscience.dataforge.meta.Meta
-import space.kscience.dataforge.misc.DFExperimental
-import space.kscience.dataforge.names.*
-import kotlin.coroutines.CoroutineContext
-import kotlin.coroutines.EmptyCoroutineContext
-import kotlin.reflect.KType
-
-
-/**
- * A stateless filtered [DataSet]
- */
-public fun <T : Any> DataSet<T>.filter(
-    predicate: (Name, Meta) -> Boolean,
-): DataSource<T> = object : DataSource<T> {
-
-    override val dataType: KType get() = this@filter.dataType
-
-    override val coroutineContext: CoroutineContext
-        get() = (this@filter as? DataSource)?.coroutineContext ?: EmptyCoroutineContext
-
-
-    override val meta: Meta get() = this@filter.meta
-
-    override fun iterator(): Iterator<NamedData<T>> = iterator {
-        for (d in this@filter) {
-            if (predicate(d.name, d.meta)) {
-                yield(d)
-            }
-        }
-    }
-
-    override fun get(name: Name): Data<T>? = this@filter.get(name)?.takeIf {
-        predicate(name, it.meta)
-    }
-
-    override val updates: Flow<Name> = this@filter.updates.filter flowFilter@{ name ->
-        val theData = this@filter[name] ?: return@flowFilter false
-        predicate(name, theData.meta)
-    }
-}
-
-/**
- * Generate a wrapper data set with a given name prefix appended to all names
- */
-public fun <T : Any> DataSet<T>.withNamePrefix(prefix: Name): DataSet<T> = if (prefix.isEmpty()) {
-    this
-} else object : DataSource<T> {
-
-    override val dataType: KType get() = this@withNamePrefix.dataType
-
-    override val coroutineContext: CoroutineContext
-        get() = (this@withNamePrefix as? DataSource)?.coroutineContext ?: EmptyCoroutineContext
-
-    override val meta: Meta get() = this@withNamePrefix.meta
-
-
-    override fun iterator(): Iterator<NamedData<T>> = iterator {
-        for (d in this@withNamePrefix) {
-            yield(d.data.named(prefix + d.name))
-        }
-    }
-
-    override fun get(name: Name): Data<T>? =
-        name.removeFirstOrNull(name)?.let { this@withNamePrefix.get(it) }
-
-    override val updates: Flow<Name> get() = this@withNamePrefix.updates.map { prefix + it }
-}
-
-/**
- * Get a subset of data starting with a given [branchName]
- */
-public fun <T : Any> DataSet<T>.branch(branchName: Name): DataSet<T> = if (branchName.isEmpty()) {
-    this
-} else object : DataSource<T> {
-    override val dataType: KType get() = this@branch.dataType
-
-    override val coroutineContext: CoroutineContext
-        get() = (this@branch as? DataSource)?.coroutineContext ?: EmptyCoroutineContext
-
-    override val meta: Meta get() = this@branch.meta
-
-    override fun iterator(): Iterator<NamedData<T>> = iterator {
-        for (d in this@branch) {
-            d.name.removeFirstOrNull(branchName)?.let { name ->
-                yield(d.data.named(name))
-            }
-        }
-    }
-
-    override fun get(name: Name): Data<T>? = this@branch.get(branchName + name)
-
-    override val updates: Flow<Name> get() = this@branch.updates.mapNotNull { it.removeFirstOrNull(branchName) }
-}
-
-public fun <T : Any> DataSet<T>.branch(branchName: String): DataSet<T> = this@branch.branch(branchName.parseAsName())
-
-@DFExperimental
-public suspend fun <T : Any> DataSet<T>.rootData(): Data<T>? = get(Name.EMPTY)
-

dataforge-data/src/commonMain/kotlin/space/kscience/dataforge/data/dataTransform.kt

@@ -1,24 +1,40 @@
 package space.kscience.dataforge.data
 
-import space.kscience.dataforge.meta.Meta
+import kotlinx.coroutines.CoroutineScope
-import space.kscience.dataforge.meta.MutableMeta
+import space.kscience.dataforge.meta.*
-import space.kscience.dataforge.meta.seal
+import space.kscience.dataforge.misc.UnsafeKType
-import space.kscience.dataforge.meta.toMutableMeta
-import space.kscience.dataforge.misc.DFInternal
 import space.kscience.dataforge.names.Name
+import space.kscience.dataforge.names.NameToken
 import kotlin.coroutines.CoroutineContext
 import kotlin.coroutines.EmptyCoroutineContext
 import kotlin.reflect.KType
 import kotlin.reflect.typeOf
 
-public data class ValueWithMeta<T>(val meta: Meta, val value: T)
+public data class ValueWithMeta<T>(val value: T, val meta: Meta)
 
-public suspend fun <T : Any> Data<T>.awaitWithMeta(): ValueWithMeta<T> = ValueWithMeta(meta, await())
+public suspend fun <T> Data<T>.awaitWithMeta(): ValueWithMeta<T> = ValueWithMeta(await(), meta)
 
-public data class NamedValueWithMeta<T>(val name: Name, val meta: Meta, val value: T)
+public data class NamedValueWithMeta<T>(val name: Name, val value: T, val meta: Meta)
 
-public suspend fun <T : Any> NamedData<T>.awaitWithMeta(): NamedValueWithMeta<T> =
+public suspend fun <T> NamedData<T>.awaitWithMeta(): NamedValueWithMeta<T> =
-    NamedValueWithMeta(name, meta, await())
+    NamedValueWithMeta(name, await(), meta)
+
+/**
+ * Lazily transform this data to another data. By convention [block] should not use external data (be pure).
+ * @param type explicit type of the resulting [Data]
+ * @param coroutineContext additional [CoroutineContext] elements used for data computation.
+ * @param meta for the resulting data. By default equals input data.
+ * @param block the transformation itself
+ */
+@UnsafeKType
+public fun <T, R> Data<T>.transform(
+    type: KType,
+    meta: Meta = this.meta,
+    coroutineContext: CoroutineContext = EmptyCoroutineContext,
+    block: suspend (T) -> R,
+): Data<R> = Data(type, meta, coroutineContext, listOf(this)) {
+    block(await())
+}
 
 
 /**
@@ -27,9 +43,9 @@ public suspend fun <T : Any> NamedData<T>.awaitWithMeta(): NamedValueWithMeta<T>
  * @param meta for the resulting data. By default equals input data.
  * @param block the transformation itself
  */
-public inline fun <T : Any, reified R : Any> Data<T>.map(
+public inline fun <T, reified R> Data<T>.transform(
-    coroutineContext: CoroutineContext = EmptyCoroutineContext,
     meta: Meta = this.meta,
+    coroutineContext: CoroutineContext = EmptyCoroutineContext,
     crossinline block: suspend (T) -> R,
 ): Data<R> = Data(meta, coroutineContext, listOf(this)) {
     block(await())
@@ -38,10 +54,10 @@ public inline fun <T : Any, reified R : Any> Data<T>.map(
 /**
  * Combine this data with the other data using [block]. See [Data::map] for other details
  */
-public inline fun <T1 : Any, T2 : Any, reified R : Any> Data<T1>.combine(
+public inline fun <T1, T2, reified R> Data<T1>.combine(
     other: Data<T2>,
-    coroutineContext: CoroutineContext = EmptyCoroutineContext,
     meta: Meta = this.meta,
+    coroutineContext: CoroutineContext = EmptyCoroutineContext,
     crossinline block: suspend (left: T1, right: T2) -> R,
 ): Data<R> = Data(meta, coroutineContext, listOf(this, other)) {
     block(await(), other.await())
@@ -50,26 +66,30 @@ public inline fun <T1 : Any, T2 : Any, reified R : Any> Data<T1>.combine(
 
 //data collection operations
 
-/**
+@PublishedApi
- * Lazily reduce a collection of [Data] to a single data.
+internal fun Iterable<Data<*>>.joinMeta(): Meta = Meta {
- */
+    var counter = 0
-public inline fun <T : Any, reified R : Any> Collection<Data<T>>.reduceToData(
+    forEach { data ->
-    coroutineContext: CoroutineContext = EmptyCoroutineContext,
+        val inputIndex = (data as? NamedData)?.name?.toString() ?: (counter++).toString()
-    meta: Meta = Meta.EMPTY,
+        val token = NameToken("data", inputIndex)
-    crossinline block: suspend (List<ValueWithMeta<T>>) -> R,
+        set(token, data.meta)
-): Data<R> = Data(
+    }
-    meta,
-    coroutineContext,
-    this
-) {
-    block(map { it.awaitWithMeta() })
 }
 
-@DFInternal
+
-public fun <K, T : Any, R : Any> Map<K, Data<T>>.reduceToData(
+@PublishedApi
+internal fun Map<*, Data<*>>.joinMeta(): Meta = Meta {
+    forEach { (key, data) ->
+        val token = NameToken("data", key.toString())
+        set(token, data.meta)
+    }
+}
+
+@UnsafeKType
+public fun <K, T, R> Map<K, Data<T>>.reduceToData(
     outputType: KType,
+    meta: Meta = joinMeta(),
     coroutineContext: CoroutineContext = EmptyCoroutineContext,
-    meta: Meta = Meta.EMPTY,
     block: suspend (Map<K, ValueWithMeta<T>>) -> R,
 ): Data<R> = Data(
     outputType,
@@ -86,9 +106,9 @@ public fun <K, T : Any, R : Any> Map<K, Data<T>>.reduceToData(
  * @param T type of the input goal
  * @param R type of the result goal
  */
-public inline fun <K, T : Any, reified R : Any> Map<K, Data<T>>.reduceToData(
+public inline fun <K, T, reified R> Map<K, Data<T>>.reduceToData(
+    meta: Meta = joinMeta(),
     coroutineContext: CoroutineContext = EmptyCoroutineContext,
-    meta: Meta = Meta.EMPTY,
     crossinline block: suspend (Map<K, ValueWithMeta<T>>) -> R,
 ): Data<R> = Data(
     meta,
@@ -100,11 +120,11 @@ public inline fun <K, T : Any, reified R : Any> Map<K, Data<T>>.reduceToData(
 
 //Iterable operations
 
-@DFInternal
+@UnsafeKType
-public inline fun <T : Any, R : Any> Iterable<Data<T>>.reduceToData(
+public inline fun <T, R> Iterable<Data<T>>.reduceToData(
     outputType: KType,
+    meta: Meta = joinMeta(),
     coroutineContext: CoroutineContext = EmptyCoroutineContext,
-    meta: Meta = Meta.EMPTY,
     crossinline transformation: suspend (Collection<ValueWithMeta<T>>) -> R,
 ): Data<R> = Data(
     outputType,
@@ -115,22 +135,22 @@ public inline fun <T : Any, R : Any> Iterable<Data<T>>.reduceToData(
     transformation(map { it.awaitWithMeta() })
 }
 
-@OptIn(DFInternal::class)
+@OptIn(UnsafeKType::class)
-public inline fun <T : Any, reified R : Any> Iterable<Data<T>>.reduceToData(
+public inline fun <T, reified R> Iterable<Data<T>>.reduceToData(
+    meta: Meta = joinMeta(),
     coroutineContext: CoroutineContext = EmptyCoroutineContext,
-    meta: Meta = Meta.EMPTY,
     crossinline transformation: suspend (Collection<ValueWithMeta<T>>) -> R,
-): Data<R> = reduceToData(typeOf<R>(), coroutineContext, meta) {
+): Data<R> = reduceToData(typeOf<R>(), meta, coroutineContext) {
     transformation(it)
 }
 
-public inline fun <T : Any, reified R : Any> Iterable<Data<T>>.foldToData(
+public inline fun <T, reified R> Iterable<Data<T>>.foldToData(
     initial: R,
+    meta: Meta = joinMeta(),
     coroutineContext: CoroutineContext = EmptyCoroutineContext,
-    meta: Meta = Meta.EMPTY,
     crossinline block: suspend (result: R, data: ValueWithMeta<T>) -> R,
 ): Data<R> = reduceToData(
-    coroutineContext, meta
+    meta, coroutineContext
 ) {
     it.fold(initial) { acc, t -> block(acc, t) }
 }
@@ -138,11 +158,11 @@ public inline fun <T : Any, reified R : Any> Iterable<Data<T>>.foldToData(
 /**
  * Transform an [Iterable] of [NamedData] to a single [Data].
  */
-@DFInternal
+@UnsafeKType
-public inline fun <T : Any, R : Any> Iterable<NamedData<T>>.reduceNamedToData(
+public inline fun <T, R> Iterable<NamedData<T>>.reduceNamedToData(
     outputType: KType,
+    meta: Meta = joinMeta(),
     coroutineContext: CoroutineContext = EmptyCoroutineContext,
-    meta: Meta = Meta.EMPTY,
     crossinline transformation: suspend (Collection<NamedValueWithMeta<T>>) -> R,
 ): Data<R> = Data(
     outputType,
@@ -153,69 +173,99 @@ public inline fun <T : Any, R : Any> Iterable<NamedData<T>>.reduceNamedToData(
     transformation(map { it.awaitWithMeta() })
 }
 
-@OptIn(DFInternal::class)
+@OptIn(UnsafeKType::class)
-public inline fun <T : Any, reified R : Any> Iterable<NamedData<T>>.reduceNamedToData(
+public inline fun <T, reified R> Iterable<NamedData<T>>.reduceNamedToData(
+    meta: Meta = joinMeta(),
     coroutineContext: CoroutineContext = EmptyCoroutineContext,
-    meta: Meta = Meta.EMPTY,
     crossinline transformation: suspend (Collection<NamedValueWithMeta<T>>) -> R,
-): Data<R> = reduceNamedToData(typeOf<R>(), coroutineContext, meta) {
+): Data<R> = reduceNamedToData(typeOf<R>(), meta, coroutineContext) {
     transformation(it)
 }
 
 /**
  * Fold a [Iterable] of named data into a single [Data]
  */
-public inline fun <T : Any, reified R : Any> Iterable<NamedData<T>>.foldNamedToData(
+public inline fun <T, reified R> Iterable<NamedData<T>>.foldNamedToData(
     initial: R,
+    meta: Meta = joinMeta(),
     coroutineContext: CoroutineContext = EmptyCoroutineContext,
-    meta: Meta = Meta.EMPTY,
     crossinline block: suspend (result: R, data: NamedValueWithMeta<T>) -> R,
 ): Data<R> = reduceNamedToData(
-    coroutineContext, meta
+    meta, coroutineContext
 ) {
     it.fold(initial) { acc, t -> block(acc, t) }
 }
 
 //DataSet operations
 
-@DFInternal
+
-public suspend fun <T : Any, R : Any> DataSet<T>.map(
+@UnsafeKType
+public fun <T, R> DataTree<T>.transformEach(
     outputType: KType,
-    coroutineContext: CoroutineContext = EmptyCoroutineContext,
+    scope: CoroutineScope,
-    metaTransform: MutableMeta.() -> Unit = {},
+    metaTransform: MutableMeta.(name: Name) -> Unit = {},
-    block: suspend (NamedValueWithMeta<T>) -> R,
+    compute: suspend (NamedValueWithMeta<T>) -> R,
-): DataTree<R> = DataTree<R>(outputType) {
+): DataTree<R> = DataTree<R>(
-    forEach {
+    outputType,
-        val newMeta = it.meta.toMutableMeta().apply(metaTransform).seal()
+    scope,
-        val d = Data(outputType, newMeta, coroutineContext, listOf(it)) {
+    initialData = asSequence().associate { namedData: NamedData<T> ->
-            block(it.awaitWithMeta())
+        val newMeta = namedData.meta.toMutableMeta().apply {
+            metaTransform(namedData.name)
+        }.seal()
+        val newData = Data(outputType, newMeta, scope.coroutineContext, listOf(namedData)) {
+            compute(namedData.awaitWithMeta())
+        }
+        namedData.name to newData
+    }
+) {
+    updates.collect { name ->
+        val data: Data<T>? = read(name)
+        if (data == null) write(name, null) else {
+            val newMeta = data.meta.toMutableMeta().apply {
+                metaTransform(name)
+            }.seal()
+            val d = Data(outputType, newMeta, scope.coroutineContext, listOf(data)) {
+                compute(NamedValueWithMeta(name, data.await(), data.meta))
+            }
+            write(name, d)
         }
-        data(it.name, d)
     }
 }
 
-@OptIn(DFInternal::class)
+@OptIn(UnsafeKType::class)
-public suspend inline fun <T : Any, reified R : Any> DataSet<T>.map(
+public inline fun <T, reified R> DataTree<T>.transformEach(
-    coroutineContext: CoroutineContext = EmptyCoroutineContext,
+    scope: CoroutineScope,
-    noinline metaTransform: MutableMeta.() -> Unit = {},
+    noinline metaTransform: MutableMeta.(name: Name) -> Unit = {},
     noinline block: suspend (NamedValueWithMeta<T>) -> R,
-): DataTree<R> = map(typeOf<R>(), coroutineContext, metaTransform, block)
+): DataTree<R> = transformEach(typeOf<R>(), scope, metaTransform, block)
 
-public inline fun <T : Any> DataSet<T>.forEach(block: (NamedData<T>) -> Unit) {
+public inline fun <T> DataTree<T>.forEach(block: (NamedData<T>) -> Unit) {
-    for (d in this) {
+    asSequence().forEach(block)
-        block(d)
+}
+
+// DataSet snapshot reduction
+
+@PublishedApi
+internal fun DataTree<*>.joinMeta(): Meta = Meta {
+    asSequence().forEach {
+        val token = NameToken("data", it.name.toString())
+        set(token, it.meta)
     }
 }
 
-public inline fun <T : Any, reified R : Any> DataSet<T>.reduceToData(
+/**
+ * Reduce current snapshot of the [DataTree] to a single [Data].
+ * Even if a tree is changed in the future, only current data set is taken.
+ */
+public inline fun <T, reified R> DataTree<T>.reduceToData(
+    meta: Meta = joinMeta(),
     coroutineContext: CoroutineContext = EmptyCoroutineContext,
-    meta: Meta = Meta.EMPTY,
     crossinline transformation: suspend (Iterable<NamedValueWithMeta<T>>) -> R,
-): Data<R> = asIterable().reduceNamedToData(coroutineContext, meta, transformation)
+): Data<R> = asSequence().asIterable().reduceNamedToData(meta, coroutineContext, transformation)
 
-public inline fun <T : Any, reified R : Any> DataSet<T>.foldToData(
+public inline fun <T, reified R> DataTree<T>.foldToData(
     initial: R,
+    meta: Meta = joinMeta(),
     coroutineContext: CoroutineContext = EmptyCoroutineContext,
-    meta: Meta = Meta.EMPTY,
     crossinline block: suspend (result: R, data: NamedValueWithMeta<T>) -> R,
-): Data<R> = asIterable().foldNamedToData(initial, coroutineContext, meta, block)
+): Data<R> = asSequence().asIterable().foldNamedToData(initial, meta, coroutineContext, block)

dataforge-data/src/commonMain/kotlin/space/kscience/dataforge/data/dataTreeBuilder.kt

@@ -0,0 +1,122 @@
+package space.kscience.dataforge.data
+
+import kotlinx.coroutines.CoroutineScope
+import kotlinx.coroutines.flow.Flow
+import kotlinx.coroutines.flow.MutableSharedFlow
+import kotlinx.coroutines.flow.SharedFlow
+import kotlinx.coroutines.flow.mapNotNull
+import kotlinx.coroutines.launch
+import kotlinx.coroutines.sync.Mutex
+import kotlinx.coroutines.sync.withLock
+import space.kscience.dataforge.misc.UnsafeKType
+import space.kscience.dataforge.names.*
+import kotlin.reflect.KType
+import kotlin.reflect.typeOf
+
+
+private class FlatDataTree<T>(
+    override val dataType: KType,
+    private val dataSet: Map<Name, Data<T>>,
+    private val sourceUpdates: SharedFlow<Name>,
+    private val prefix: Name,
+) : DataTree<T> {
+    override val data: Data<T>? get() = dataSet[prefix]
+    override val items: Map<NameToken, FlatDataTree<T>>
+        get() = dataSet.keys
+            .filter { it.startsWith(prefix) && it.length > prefix.length }
+            .map { it.tokens[prefix.length] }
+            .associateWith { FlatDataTree(dataType, dataSet, sourceUpdates, prefix + it) }
+
+    override fun read(name: Name): Data<T>? = dataSet[prefix + name]
+
+    override val updates: Flow<Name> = sourceUpdates.mapNotNull { update ->
+        update.removeFirstOrNull(prefix)
+    }
+}
+
+/**
+ * A builder for [DataTree].
+ */
+private class DataTreeBuilder<T>(
+    private val type: KType,
+    initialData: Map<Name, Data<T>> = emptyMap(),
+) : DataSink<T> {
+
+    private val map = HashMap<Name, Data<T>>(initialData)
+
+    private val mutex = Mutex()
+
+    private val updatesFlow = MutableSharedFlow<Name>()
+
+
+    override suspend fun write(name: Name, data: Data<T>?) {
+        mutex.withLock {
+            if (data == null) {
+                map.remove(name)
+            } else {
+                map[name] = data
+            }
+        }
+        updatesFlow.emit(name)
+    }
+
+    fun build(): DataTree<T> = FlatDataTree(type, map, updatesFlow, Name.EMPTY)
+}
+
+/**
+ * Create a static [DataTree]
+ */
+@UnsafeKType
+public fun <T> DataTree(
+    dataType: KType,
+    scope: CoroutineScope,
+    initialData: Map<Name, Data<T>> = emptyMap(),
+    updater: suspend DataSink<T>.() -> Unit,
+): DataTree<T> = DataTreeBuilder<T>(dataType, initialData).apply {
+    scope.launch(GoalExecutionRestriction(GoalExecutionRestrictionPolicy.ERROR)) {
+        updater()
+    }
+}.build()
+
+/**
+ * Create and a data tree.
+ */
+@OptIn(UnsafeKType::class)
+public inline fun <reified T> DataTree(
+    scope: CoroutineScope,
+    initialData: Map<Name, Data<T>> = emptyMap(),
+    noinline updater: suspend DataSink<T>.() -> Unit,
+): DataTree<T> = DataTree(typeOf<T>(), scope, initialData, updater)
+
+@UnsafeKType
+public fun <T> DataTree(type: KType, data: Map<Name, Data<T>>): DataTree<T> =
+    DataTreeBuilder(type, data).build()
+
+@OptIn(UnsafeKType::class)
+public inline fun <reified T> DataTree(data: Map<Name, Data<T>>): DataTree<T> =
+    DataTree(typeOf<T>(), data)
+
+/**
+ * Represent this flat data map as a [DataTree] without copying it
+ */
+@UnsafeKType
+public fun <T> Map<Name, Data<T>>.asTree(type: KType): DataTree<T> =
+    DataTreeBuilder(type, this).build()
+
+/**
+ * Represent this flat data map as a [DataTree] without copying it
+ */
+@OptIn(UnsafeKType::class)
+public inline fun <reified T> Map<Name, Data<T>>.asTree(): DataTree<T> = asTree(typeOf<T>())
+
+
+@UnsafeKType
+public fun <T> Sequence<NamedData<T>>.toTree(type: KType): DataTree<T> =
+    DataTreeBuilder(type, associate { it.name to it }).build()
+
+
+/**
+ * Collect a sequence of [NamedData] to a [DataTree]
+ */
+@OptIn(UnsafeKType::class)
+public inline fun <reified T> Sequence<NamedData<T>>.toTree(): DataTree<T> = toTree(typeOf<T>())

dataforge-data/src/commonTest/kotlin/space/kscience/dataforge/data/DataTreeBuilderTest.kt

@@ -0,0 +1,73 @@
+package space.kscience.dataforge.data
+
+import kotlinx.coroutines.Job
+import kotlinx.coroutines.launch
+import kotlinx.coroutines.test.runTest
+import space.kscience.dataforge.names.asName
+import kotlin.test.Test
+import kotlin.test.assertEquals
+import kotlin.time.Duration.Companion.milliseconds
+
+
+internal class DataTreeBuilderTest {
+    @Test
+    fun testTreeBuild() = runTest(timeout = 500.milliseconds) {
+        val node = DataTree.static<Any> {
+            node("primary") {
+                value("a", "a")
+                value("b", "b")
+            }
+            value("c.d", "c.d")
+            value("c.f", "c.f")
+        }
+        assertEquals("a", node["primary.a"]?.await())
+        assertEquals("b", node["primary.b"]?.await())
+        assertEquals("c.d", node["c.d"]?.await())
+        assertEquals("c.f", node["c.f"]?.await())
+
+    }
+
+    @Test
+    fun testDataUpdate() = runTest(timeout = 500.milliseconds) {
+        val updateData = DataTree.static<Any> {
+            data("a", Data.wrapValue("a"))
+            data("b", Data.wrapValue("b"))
+        }
+
+        val node = DataTree.static<Any> {
+            node("primary") {
+                value("a", "a")
+                value("b", "b")
+            }
+            value("root", "root")
+            node("update", updateData)
+        }
+
+        assertEquals("a", node["update.a"]?.await())
+        assertEquals("a", node["primary.a"]?.await())
+    }
+
+    @Test
+    fun testDynamicUpdates() = runTest(timeout = 500.milliseconds) {
+        var job: Job? = null
+
+        val subNode = MutableDataTree<Int>()
+
+        val rootNode = MutableDataTree<Int>() {
+            job = launch {
+                writeAllAndWatch(subNode, "sub".asName())
+            }
+        }
+
+        repeat(10) {
+            subNode.writeValue("value[$it]", it)
+        }
+
+        assertEquals(9, subNode.awaitData("value[9]").await())
+        assertEquals(8, subNode.awaitData("value[8]").await())
+        assertEquals(9, rootNode.awaitData("sub.value[9]").await())
+        assertEquals(8, rootNode.awaitData("sub.value[8]").await())
+        println("finished")
+        job?.cancel()
+    }
+}

dataforge-data/src/jvmMain/kotlin/space/kscience/dataforge/data/actionInContext.kt

@@ -1,2 +0,0 @@
-package space.kscience.dataforge.data
-

dataforge-data/src/jvmMain/kotlin/space/kscience/dataforge/data/dataFilterJvm.kt

@@ -1,85 +0,0 @@
-package space.kscience.dataforge.data
-
-import kotlinx.coroutines.flow.Flow
-import kotlinx.coroutines.flow.filter
-import space.kscience.dataforge.meta.Meta
-import space.kscience.dataforge.misc.DFExperimental
-import space.kscience.dataforge.names.Name
-import kotlin.coroutines.CoroutineContext
-import kotlin.coroutines.EmptyCoroutineContext
-import kotlin.reflect.KType
-import kotlin.reflect.full.isSubtypeOf
-import kotlin.reflect.typeOf
-
-
-/**
- * Cast the node to given type if the cast is possible or return null
- */
-@Suppress("UNCHECKED_CAST")
-private fun <R : Any> Data<*>.castOrNull(type: KType): Data<R>? =
-    if (!this.type.isSubtypeOf(type)) {
-        null
-    } else {
-        object : Data<R> by (this as Data<R>) {
-            override val type: KType = type
-        }
-    }
-
-/**
- * Select all data matching given type and filters. Does not modify paths
- *
- * @param predicate addition filtering condition based on item name and meta. By default, accepts all
- */
-@OptIn(DFExperimental::class)
-public fun <R : Any> DataSet<*>.filterByType(
-    type: KType,
-    predicate: (name: Name, meta: Meta) -> Boolean = { _, _ -> true },
-): DataSource<R> = object : DataSource<R> {
-    override val dataType = type
-
-    override val coroutineContext: CoroutineContext
-        get() = (this@filterByType as? DataSource)?.coroutineContext ?: EmptyCoroutineContext
-
-    override val meta: Meta get() = this@filterByType.meta
-
-    private fun checkDatum(name: Name, datum: Data<*>): Boolean = datum.type.isSubtypeOf(type)
-            && predicate(name, datum.meta)
-
-    override fun iterator(): Iterator<NamedData<R>> = iterator {
-        for(d in this@filterByType){
-            if(checkDatum(d.name,d.data)){
-                @Suppress("UNCHECKED_CAST")
-                yield(d as NamedData<R>)
-            }
-        }
-    }
-
-    override fun get(name: Name): Data<R>? = this@filterByType[name]?.let { datum ->
-        if (checkDatum(name, datum)) datum.castOrNull(type) else null
-    }
-
-    override val updates: Flow<Name> = this@filterByType.updates.filter { name ->
-        get(name)?.let { datum ->
-            checkDatum(name, datum)
-        } ?: false
-    }
-}
-
-/**
- * Select a single datum of the appropriate type
- */
-public inline fun <reified R : Any> DataSet<*>.filterByType(
-    noinline predicate: (name: Name, meta: Meta) -> Boolean = { _, _ -> true },
-): DataSet<R> = filterByType(typeOf<R>(), predicate)
-
-/**
- * Select a single datum if it is present and of given [type]
- */
-public fun <R : Any> DataSet<*>.getByType(type: KType, name: Name): NamedData<R>? =
-    get(name)?.castOrNull<R>(type)?.named(name)
-
-public inline fun <reified R : Any> DataSet<*>.getByType(name: Name): NamedData<R>? =
-    this@getByType.getByType(typeOf<R>(), name)
-
-public inline fun <reified R : Any> DataSet<*>.getByType(name: String): NamedData<R>? =
-    this@getByType.getByType(typeOf<R>(), Name.parse(name))

dataforge-data/src/jvmMain/kotlin/space/kscience/dataforge/data/dataSetBuilderInContext.kt

@@ -1,40 +1,29 @@
+@file:Suppress("CONTEXT_RECEIVERS_DEPRECATED")
+
 package space.kscience.dataforge.data
 
-import kotlinx.coroutines.CoroutineScope
-import kotlinx.coroutines.Job
-import kotlinx.coroutines.launch
 import space.kscience.dataforge.names.Name
-import space.kscience.dataforge.names.plus
 
 
 /**
  * Append data to node
  */
-context(DataSetBuilder<T>) public infix fun <T : Any> String.put(data: Data<T>): Unit =
+context(DataSink<T>)
-    data(Name.parse(this), data)
+public suspend infix fun <T : Any> String.put(data: Data<T>): Unit =
+    write(Name.parse(this), data)
 
 /**
  * Append node
  */
-context(DataSetBuilder<T>) public infix fun <T : Any> String.put(dataSet: DataSet<T>): Unit =
+context(DataSink<T>)
-    node(Name.parse(this), dataSet)
+public suspend infix fun <T : Any> String.putAll(dataSet: DataTree<T>): Unit =
+    writeAll(this, dataSet)
 
 /**
  * Build and append node
  */
-context(DataSetBuilder<T>) public infix fun <T : Any> String.put(
+context(DataSink<T>)
-    block: DataSetBuilder<T>.() -> Unit,
+public infix fun <T : Any> String.putAll(
-): Unit = node(Name.parse(this), block)
+    block: DataSink<T>.() -> Unit,
+): Unit = writeAll(Name.parse(this), block)
 
-/**
- * Copy given data set and mirror its changes to this [DataTreeBuilder] in [this@setAndObserve]. Returns an update [Job]
- */
-context(DataSetBuilder<T>) public fun <T : Any> CoroutineScope.setAndWatch(
-    name: Name,
-    dataSet: DataSet<T>,
-): Job = launch {
-    node(name, dataSet)
-    dataSet.updates.collect { nameInBranch ->
-        data(name + nameInBranch, dataSet.get(nameInBranch))
-    }
-}

dataforge-data/src/jvmTest/kotlin/space/kscience/dataforge/data/ActionsTest.kt

@@ -1,50 +1,49 @@
 package space.kscience.dataforge.data
 
 import kotlinx.coroutines.ExperimentalCoroutinesApi
-import kotlinx.coroutines.delay
 import kotlinx.coroutines.test.runTest
-import org.junit.jupiter.api.Test
 import space.kscience.dataforge.actions.Action
 import space.kscience.dataforge.actions.invoke
-import space.kscience.dataforge.actions.map
+import space.kscience.dataforge.actions.mapping
 import space.kscience.dataforge.misc.DFExperimental
+import kotlin.test.Test
 import kotlin.test.assertEquals
+import kotlin.time.Duration.Companion.milliseconds
 
 @OptIn(DFExperimental::class, ExperimentalCoroutinesApi::class)
 internal class ActionsTest {
     @Test
-    fun testStaticMapAction() = runTest {
+    fun testStaticMapAction() = runTest(timeout = 200.milliseconds) {
-        val data: DataTree<Int> = DataTree {
+        val plusOne = Action.mapping<Int, Int> {
+            result { it + 1 }
+        }
+
+        val data: DataTree<Int> = DataTree.static {
             repeat(10) {
-                static(it.toString(), it)
+                value(it.toString(), it)
             }
         }
 
-        val plusOne = Action.map<Int, Int> {
-            result { it + 1 }
-        }
         val result = plusOne(data)
-        assertEquals(2, result["1"]?.await())
+
+        assertEquals(5, result.awaitData("4").await())
     }
 
     @Test
-    fun testDynamicMapAction() = runTest {
+    fun testDynamicMapAction() = runTest(timeout = 200.milliseconds) {
-        val data: DataSourceBuilder<Int> = DataSource()
+        val plusOne = Action.mapping<Int, Int> {
-
-        val plusOne = Action.map<Int, Int> {
             result { it + 1 }
         }
 
-        val result = plusOne(data)
+        val source: MutableDataTree<Int> = MutableDataTree()
+
+        val result: DataTree<Int> = plusOne(source)
 
         repeat(10) {
-            data.static(it.toString(), it)
+            source.writeValue(it.toString(), it)
         }
 
-        delay(20)
+        assertEquals(5, result.awaitData("4").await())
-
-        assertEquals(2, result["1"]?.await())
-        data.close()
     }
 
 }

dataforge-data/src/jvmTest/kotlin/space/kscience/dataforge/data/DataTreeBuilderTest.kt

@@ -1,91 +0,0 @@
-package space.kscience.dataforge.data
-
-import kotlinx.coroutines.*
-import space.kscience.dataforge.misc.DFExperimental
-import space.kscience.dataforge.names.asName
-import kotlin.test.Test
-import kotlin.test.assertEquals
-
-
-internal class DataTreeBuilderTest {
-    @Test
-    fun testTreeBuild() = runBlocking {
-        val node = DataTree<Any> {
-            "primary" put {
-                static("a", "a")
-                static("b", "b")
-            }
-            static("c.d", "c.d")
-            static("c.f", "c.f")
-        }
-        runBlocking {
-            assertEquals("a", node["primary.a"]?.await())
-            assertEquals("b", node["primary.b"]?.await())
-            assertEquals("c.d", node["c.d"]?.await())
-            assertEquals("c.f", node["c.f"]?.await())
-        }
-    }
-
-    @OptIn(DFExperimental::class)
-    @Test
-    fun testDataUpdate() = runBlocking {
-        val updateData: DataTree<Any> = DataTree {
-            "update" put {
-                "a" put Data.static("a")
-                "b" put Data.static("b")
-            }
-        }
-
-        val node = DataTree<Any> {
-            "primary" put {
-                static("a", "a")
-                static("b", "b")
-            }
-            static("root", "root")
-            populateFrom(updateData)
-        }
-
-        runBlocking {
-            assertEquals("a", node["update.a"]?.await())
-            assertEquals("a", node["primary.a"]?.await())
-        }
-    }
-
-    @Test
-    fun testDynamicUpdates() = runBlocking {
-        try {
-            lateinit var updateJob: Job
-            supervisorScope {
-                val subNode = DataSource<Int> {
-                    updateJob = launch {
-                        repeat(10) {
-                            delay(10)
-                            static("value", it)
-                        }
-                        delay(10)
-                    }
-                }
-                launch {
-                    subNode.updatesWithData.collect {
-                        println(it)
-                    }
-                }
-                val rootNode = DataSource<Int> {
-                    setAndWatch("sub".asName(), subNode)
-                }
-
-                launch {
-                    rootNode.updatesWithData.collect {
-                        println(it)
-                    }
-                }
-                updateJob.join()
-                assertEquals(9, rootNode["sub.value"]?.await())
-                cancel()
-            }
-        } catch (t: Throwable) {
-            if (t !is CancellationException) throw t
-        }
-
-    }
-}

dataforge-io/README.md

@@ -2,22 +2,29 @@
 
 IO module
 
+## Features
+
+ - [IO format](src/commonMain/kotlin/space/kscience/dataforge/io/IOFormat.kt) : A generic API for reading something from binary representation and writing it to Binary.
+ - [Binary](src/commonMain/kotlin/space/kscience/dataforge/io/Binary.kt) : Multi-read random access binary.
+ - [Envelope](src/commonMain/kotlin/space/kscience/dataforge/io/Envelope.kt) : API and implementations for combined data and metadata format.
+ - [Tagged envelope](src/commonMain/kotlin/space/kscience/dataforge/io/TaggedEnvelope.kt) : Implementation for binary-friendly envelope format with machine readable tag and forward size declaration.
+ - [Tagged envelope](src/commonMain/kotlin/space/kscience/dataforge/io/TaglessEnvelope.kt) : Implementation for text-friendly envelope format with text separators for sections.
+
+
 ## Usage
 
 ## Artifact:
 
-The Maven coordinates of this project are `space.kscience:dataforge-io:0.7.0`.
+The Maven coordinates of this project are `space.kscience:dataforge-io:0.10.0`.
 
 **Gradle Kotlin DSL:**
 ```kotlin
 repositories {
     maven("https://repo.kotlin.link")
-    //uncomment to access development builds
-    //maven("https://maven.pkg.jetbrains.space/spc/p/sci/dev")
     mavenCentral()
 }
 
 dependencies {
-    implementation("space.kscience:dataforge-io:0.7.0")
+    implementation("space.kscience:dataforge-io:0.10.0")
 }
 ```

dataforge-io/build.gradle.kts

@@ -4,12 +4,13 @@ plugins {
 
 description = "IO module"
 
-val ioVersion = "0.2.1"
+val ioVersion = "0.6.0"
 
 kscience {
     jvm()
     js()
     native()
+    wasm()
     useSerialization()
     useSerialization(sourceSet = space.kscience.gradle.DependencySourceSet.TEST) {
         cbor()
@@ -21,6 +22,60 @@ kscience {
     }
 }
 
-readme{
+readme {
     maturity = space.kscience.gradle.Maturity.EXPERIMENTAL
+
+    description = """
+        Serialization foundation for Meta objects and Envelope processing.
+    """.trimIndent()
+
+    feature(
+        "io-format",
+        ref = "src/commonMain/kotlin/space/kscience/dataforge/io/IOFormat.kt",
+        name = "IO format"
+    ) {
+        """
+            A generic API for reading something from binary representation and writing it to Binary.
+            
+            Similar to KSerializer, but without schema.
+        """.trimIndent()
+    }
+
+    feature(
+        "binary",
+        ref = "src/commonMain/kotlin/space/kscience/dataforge/io/Binary.kt",
+        name = "Binary"
+    ) {
+        "Multi-read random access binary."
+    }
+
+    feature(
+        "envelope",
+        ref = "src/commonMain/kotlin/space/kscience/dataforge/io/Envelope.kt",
+        name = "Envelope"
+    ) {
+        """
+            API and implementations for combined data and metadata format.
+        """.trimIndent()
+    }
+
+    feature(
+        "envelope.tagged",
+        ref = "src/commonMain/kotlin/space/kscience/dataforge/io/TaggedEnvelope.kt",
+        name = "Tagged envelope"
+    ) {
+        """
+            Implementation for binary-friendly envelope format with machine readable tag and forward size declaration.
+        """.trimIndent()
+    }
+
+    feature(
+        "envelope.tagless",
+        ref = "src/commonMain/kotlin/space/kscience/dataforge/io/TaglessEnvelope.kt",
+        name = "Tagged envelope"
+    ) {
+        """
+            Implementation for text-friendly envelope format with text separators for sections.
+        """.trimIndent()
+    }
 }

dataforge-io/dataforge-io-proto/README.md

@@ -0,0 +1,21 @@
+# Module dataforge-io-proto
+
+ProtoBuf meta IO
+
+## Usage
+
+## Artifact:
+
+The Maven coordinates of this project are `space.kscience:dataforge-io-proto:0.10.0`.
+
+**Gradle Kotlin DSL:**
+```kotlin
+repositories {
+    maven("https://repo.kotlin.link")
+    mavenCentral()
+}
+
+dependencies {
+    implementation("space.kscience:dataforge-io-proto:0.10.0")
+}
+```

dataforge-io/dataforge-io-proto/build.gradle.kts

@@ -0,0 +1,33 @@
+plugins {
+    id("space.kscience.gradle.mpp")
+    id("com.squareup.wire") version "4.9.9"
+}
+
+description = "ProtoBuf meta IO"
+
+kscience {
+    jvm()
+//    js()
+    dependencies {
+        api(projects.dataforgeIo)
+        api("com.squareup.wire:wire-runtime:4.9.9")
+    }
+    useSerialization {
+        protobuf()
+    }
+}
+
+wire {
+    kotlin {
+        sourcePath {
+            srcDir("src/commonMain/proto")
+        }
+    }
+}
+
+readme {
+    maturity = space.kscience.gradle.Maturity.PROTOTYPE
+    description = """
+        ProtoBuf Meta representation
+    """.trimIndent()
+}

dataforge-io/dataforge-io-proto/src/commonMain/kotlin/ProtoEnvelopeFormat.kt

@@ -0,0 +1,32 @@
+package pace.kscience.dataforge.io.proto
+
+import kotlinx.io.Sink
+import kotlinx.io.Source
+import kotlinx.io.readByteArray
+import okio.ByteString
+import okio.ByteString.Companion.toByteString
+import space.kscience.dataforge.io.Envelope
+import space.kscience.dataforge.io.EnvelopeFormat
+import space.kscience.dataforge.io.asBinary
+import space.kscience.dataforge.io.proto.ProtoEnvelope
+import space.kscience.dataforge.io.toByteArray
+import space.kscience.dataforge.meta.Meta
+
+
+public object ProtoEnvelopeFormat : EnvelopeFormat {
+    override fun readFrom(source: Source): Envelope {
+        val protoEnvelope = ProtoEnvelope.ADAPTER.decode(source.readByteArray())
+        return Envelope(
+            meta = protoEnvelope.meta?.let { ProtoMetaWrapper(it) } ?: Meta.EMPTY,
+            data = protoEnvelope.dataBytes.toByteArray().asBinary()
+        )
+    }
+
+    override fun writeTo(sink: Sink, obj: Envelope) {
+        val protoEnvelope = ProtoEnvelope(
+            obj.meta.toProto(),
+            obj.data?.toByteArray()?.toByteString() ?: ByteString.EMPTY
+        )
+        sink.write(ProtoEnvelope.ADAPTER.encode(protoEnvelope))
+    }
+}

dataforge-io/dataforge-io-proto/src/commonMain/kotlin/ProtoMetaFormat.kt

@@ -0,0 +1,76 @@
+package pace.kscience.dataforge.io.proto
+
+import kotlinx.io.Sink
+import kotlinx.io.Source
+import kotlinx.io.readByteArray
+import space.kscience.dataforge.io.MetaFormat
+import space.kscience.dataforge.io.proto.ProtoMeta
+import space.kscience.dataforge.meta.*
+import space.kscience.dataforge.meta.descriptors.MetaDescriptor
+import space.kscience.dataforge.names.NameToken
+
+internal class ProtoMetaWrapper(private val proto: ProtoMeta) : Meta {
+
+    private fun ProtoMeta.ProtoValue.toValue(): Value? = when {
+        stringValue != null -> stringValue.asValue()
+        booleanValue != null -> booleanValue.asValue()
+        doubleValue != null -> doubleValue.asValue()
+        floatValue != null -> floatValue.asValue()
+        int32Value != null -> int32Value.asValue()
+        int64Value != null -> int64Value.asValue()
+        bytesValue != null -> bytesValue.toByteArray().asValue()
+        listValue != null -> listValue.values.mapNotNull { it.toValue() }.asValue()
+        float64ListValue != null -> float64ListValue.values.map { it.asValue() }.asValue()
+        else -> null
+    }
+
+    override val value: Value?
+        get() = proto.protoValue?.toValue()
+
+
+    override val items: Map<NameToken, Meta>
+        get() = proto.items.entries.associate { NameToken.parse(it.key) to ProtoMetaWrapper(it.value) }
+
+    override fun toString(): String = Meta.toString(this)
+
+    override fun equals(other: Any?): Boolean = Meta.equals(this, other as? Meta)
+
+    override fun hashCode(): Int = Meta.hashCode(this)
+}
+
+internal fun Meta.toProto(): ProtoMeta {
+
+
+    fun Value.toProto(): ProtoMeta.ProtoValue = when (type) {
+        ValueType.NULL -> ProtoMeta.ProtoValue()
+
+        ValueType.NUMBER -> when (value) {
+            is Int, is Short, is Byte -> ProtoMeta.ProtoValue(int32Value = int)
+            is Long -> ProtoMeta.ProtoValue(int64Value = long)
+            is Float -> ProtoMeta.ProtoValue(floatValue = float)
+            else -> {
+//                LoggerFactory.getLogger(ProtoMeta::class.java)
+//                    .warn("Unknown number type ${value} encoded as Double")
+                ProtoMeta.ProtoValue(doubleValue = double)
+            }
+        }
+
+        ValueType.STRING -> ProtoMeta.ProtoValue(stringValue = string)
+        ValueType.BOOLEAN -> ProtoMeta.ProtoValue(booleanValue = boolean)
+        ValueType.LIST -> ProtoMeta.ProtoValue(listValue = ProtoMeta.ProtoValueList(list.map { it.toProto() }))
+    }
+
+    return ProtoMeta(
+        protoValue = value?.toProto(),
+        items.entries.associate { it.key.toString() to it.value.toProto() }
+    )
+}
+
+public object ProtoMetaFormat : MetaFormat {
+    override fun writeMeta(sink: Sink, meta: Meta, descriptor: MetaDescriptor?) {
+        sink.write(ProtoMeta.ADAPTER.encode(meta.toProto()))
+    }
+
+    override fun readMeta(source: Source, descriptor: MetaDescriptor?): Meta =
+        ProtoMetaWrapper(ProtoMeta.ADAPTER.decode(source.readByteArray()))
+}

dataforge-io/dataforge-io-proto/src/commonMain/proto/meta.proto

@@ -0,0 +1,35 @@
+syntax = "proto3";
+package space.kscience.dataforge.io.proto;
+
+message ProtoMeta {
+  message ProtoValue {
+    oneof value {
+      string stringValue = 2;
+      bool booleanValue = 3;
+      double doubleValue = 4;
+      float floatValue = 5;
+      int32 int32Value = 6;
+      int64 int64Value = 7;
+      bytes bytesValue = 8;
+      ProtoValueList listValue = 9;
+      Float64List float64ListValue = 10;
+    }
+  }
+
+  message ProtoValueList{
+    repeated ProtoValue values = 1;
+  }
+
+  message Float64List{
+    repeated double values = 1 [packed=true];
+  }
+
+  ProtoValue protoValue = 1;
+
+  map<string, ProtoMeta> items = 2;
+}
+
+message ProtoEnvelope{
+  ProtoMeta meta = 1;
+  bytes dataBytes = 2;
+}

dataforge-io/dataforge-io-proto/src/commonTest/kotlin/ProtoBufTest.kt

@@ -0,0 +1,83 @@
+package pace.kscience.dataforge.io.proto
+
+import kotlinx.io.writeString
+import space.kscience.dataforge.io.Envelope
+import space.kscience.dataforge.io.toByteArray
+import space.kscience.dataforge.meta.Meta
+import space.kscience.dataforge.meta.asValue
+import space.kscience.dataforge.meta.get
+import kotlin.test.Test
+import kotlin.test.assertContentEquals
+import kotlin.test.assertEquals
+
+class ProtoBufTest {
+
+    @Test
+    fun testProtoBufMetaFormat(){
+        val meta = Meta {
+            "a" put 22
+            "node" put {
+                "b" put "DDD"
+                "c" put 11.1
+                "d" put {
+                    "d1" put {
+                        "d11" put "aaa"
+                        "d12" put "bbb"
+                    }
+                    "d2" put 2
+                }
+                "array" put doubleArrayOf(1.0, 2.0, 3.0)
+                "array2d" put listOf(
+                    doubleArrayOf(1.0, 2.0, 3.0).asValue(),
+                    doubleArrayOf(1.0, 2.0, 3.0).asValue()
+                ).asValue()
+            }
+        }
+        val buffer = kotlinx.io.Buffer()
+        ProtoMetaFormat.writeTo(buffer,meta)
+        val result = ProtoMetaFormat.readFrom(buffer)
+
+//        println(result["a"]?.value)
+
+        meta.items.keys.forEach {
+            assertEquals(meta[it],result[it],"${meta[it]} != ${result[it]}")
+        }
+
+        assertEquals(meta, result)
+    }
+
+    @Test
+    fun testProtoBufEnvelopeFormat(){
+        val envelope = Envelope{
+            meta {
+                "a" put 22
+                "node" put {
+                    "b" put "DDD"
+                    "c" put 11.1
+                    "d" put {
+                        "d1" put {
+                            "d11" put "aaa"
+                            "d12" put "bbb"
+                        }
+                        "d2" put 2
+                    }
+                    "array" put doubleArrayOf(1.0, 2.0, 3.0)
+                    "array2d" put listOf(
+                        doubleArrayOf(1.0, 2.0, 3.0).asValue(),
+                        doubleArrayOf(1.0, 2.0, 3.0).asValue()
+                    ).asValue()
+                }
+            }
+            data {
+                writeString("Hello world!")
+            }
+        }
+
+        val buffer = kotlinx.io.Buffer()
+        ProtoEnvelopeFormat.writeTo(buffer,envelope)
+        val result = ProtoEnvelopeFormat.readFrom(buffer)
+
+        assertEquals(envelope.meta, result.meta)
+        assertContentEquals(envelope.data?.toByteArray(), result.data?.toByteArray())
+    }
+}

dataforge-io/dataforge-io-proto/src/jvmMain/kotlin/performanceComparison.kt

@@ -0,0 +1,51 @@
+package pace.kscience.dataforge.io.proto
+
+import kotlinx.io.writeString
+import space.kscience.dataforge.io.Envelope
+import space.kscience.dataforge.meta.asValue
+import kotlin.concurrent.thread
+import kotlin.time.measureTime
+
+public fun main() {
+    val envelope = Envelope {
+        meta {
+            "a" put 22
+            "node" put {
+                "b" put "DDD"
+                "c" put 11.1
+                "d" put {
+                    "d1" put {
+                        "d11" put "aaa"
+                        "d12" put "bbb"
+                    }
+                    "d2" put 2
+                }
+                "array" put doubleArrayOf(1.0, 2.0, 3.0)
+                "array2d" put listOf(
+                    doubleArrayOf(1.0, 2.0, 3.0).asValue(),
+                    doubleArrayOf(1.0, 2.0, 3.0).asValue()
+                ).asValue()
+            }
+        }
+        data {
+            writeString("Hello world!")
+        }
+    }
+
+    val format = ProtoEnvelopeFormat
+
+    measureTime {
+        val threads = List(100) {
+            thread {
+                repeat(100000) {
+                    val buffer = kotlinx.io.Buffer()
+                    format.writeTo(buffer, envelope)
+//                    println(buffer.size)
+                    val r = format.readFrom(buffer)
+                }
+            }
+        }
+
+        threads.forEach { it.join() }
+    }.also { println(it) }
+}

dataforge-io/dataforge-io-yaml/README.md

@@ -6,18 +6,16 @@ YAML meta IO
 
 ## Artifact:
 
-The Maven coordinates of this project are `space.kscience:dataforge-io-yaml:0.7.0`.
+The Maven coordinates of this project are `space.kscience:dataforge-io-yaml:0.10.0`.
 
 **Gradle Kotlin DSL:**
 ```kotlin
 repositories {
     maven("https://repo.kotlin.link")
-    //uncomment to access development builds
-    //maven("https://maven.pkg.jetbrains.space/spc/p/sci/dev")
     mavenCentral()
 }
 
 dependencies {
-    implementation("space.kscience:dataforge-io-yaml:0.7.0")
+    implementation("space.kscience:dataforge-io-yaml:0.10.0")
 }
 ```

dataforge-io/dataforge-io-yaml/build.gradle.kts

@@ -11,14 +11,14 @@ kscience {
     dependencies {
         api(projects.dataforgeIo)
     }
-    useSerialization{
+    useSerialization {
         yamlKt()
     }
 }
 
-readme{
+readme {
     maturity = space.kscience.gradle.Maturity.PROTOTYPE
-    description ="""
+    description = """
         YAML meta converters and Front Matter envelope format
     """.trimIndent()
 }

dataforge-io/src/commonMain/kotlin/space/kscience/dataforge/io/Binary.kt

@@ -71,6 +71,11 @@ internal class ByteArrayBinary(
 
     override fun view(offset: Int, binarySize: Int): ByteArrayBinary =
         ByteArrayBinary(array, start + offset, binarySize)
+
+    override fun toString(): String =
+        "ByteArrayBinary(array=$array, start=$start, size=$size)"
+
+
 }
 
 public fun ByteArray.asBinary(): Binary = ByteArrayBinary(this)

dataforge-io/src/commonMain/kotlin/space/kscience/dataforge/io/Envelope.kt

@@ -2,7 +2,6 @@ package space.kscience.dataforge.io
 
 import space.kscience.dataforge.meta.Laminate
 import space.kscience.dataforge.meta.Meta
-import space.kscience.dataforge.meta.get
 import space.kscience.dataforge.meta.string
 import space.kscience.dataforge.names.Name
 import space.kscience.dataforge.names.asName
@@ -34,7 +33,9 @@ public interface Envelope {
     }
 }
 
-internal class SimpleEnvelope(override val meta: Meta, override val data: Binary?) : Envelope
+internal class SimpleEnvelope(override val meta: Meta, override val data: Binary?) : Envelope{
+    override fun toString(): String  = "Envelope(meta=$meta, data=$data)"
+}
 
 public fun Envelope(meta: Meta, data: Binary?): Envelope = SimpleEnvelope(meta, data)
 

dataforge-io/src/commonMain/kotlin/space/kscience/dataforge/io/EnvelopeFormat.kt

@@ -4,20 +4,17 @@ import kotlinx.io.Source
 import space.kscience.dataforge.context.Context
 import space.kscience.dataforge.io.EnvelopeFormatFactory.Companion.ENVELOPE_FORMAT_TYPE
 import space.kscience.dataforge.meta.Meta
-import space.kscience.dataforge.misc.DfId
+import space.kscience.dataforge.misc.DfType
 import space.kscience.dataforge.names.Name
 import space.kscience.dataforge.names.asName
 import kotlin.reflect.KType
 import kotlin.reflect.typeOf
 
-public interface EnvelopeFormat : IOFormat<Envelope> {
+public interface EnvelopeFormat : IOFormat<Envelope>
-
-    override val type: KType get() = typeOf<Envelope>()
-}
 
 public fun EnvelopeFormat.read(input: Source): Envelope = readFrom(input)
 
-@DfId(ENVELOPE_FORMAT_TYPE)
+@DfType(ENVELOPE_FORMAT_TYPE)
 public interface EnvelopeFormatFactory : IOFormatFactory<Envelope>, EnvelopeFormat {
     override val type: KType get() = typeOf<Envelope>()
 

dataforge-io/src/commonMain/kotlin/space/kscience/dataforge/io/EnvelopeParts.kt

@@ -11,6 +11,7 @@ import space.kscience.dataforge.io.PartDescriptor.Companion.PARTS_KEY
 import space.kscience.dataforge.io.PartDescriptor.Companion.SEPARATOR_KEY
 import space.kscience.dataforge.meta.*
 import space.kscience.dataforge.names.asName
+import space.kscience.dataforge.names.getIndexedList
 import space.kscience.dataforge.names.plus
 
 private class PartDescriptor : Scheme() {
@@ -84,7 +85,7 @@ public fun EnvelopeBuilder.envelopes(
 public fun Envelope.parts(): EnvelopeParts {
     if (data == null) return emptyList()
     //TODO add zip folder reader
-    val parts = meta.getIndexed(PARTS_KEY).values.map {
+    val parts = meta.getIndexedList(PARTS_KEY).map {
         PartDescriptor.read(it)
     }
     return if (parts.isEmpty()) {

dataforge-io/src/commonMain/kotlin/space/kscience/dataforge/io/IOFormat.kt

@@ -7,7 +7,7 @@ import space.kscience.dataforge.context.Context
 import space.kscience.dataforge.context.Factory
 import space.kscience.dataforge.io.IOFormatFactory.Companion.IO_FORMAT_TYPE
 import space.kscience.dataforge.meta.Meta
-import space.kscience.dataforge.misc.DfId
+import space.kscience.dataforge.misc.DfType
 import space.kscience.dataforge.misc.Named
 import space.kscience.dataforge.names.Name
 import space.kscience.dataforge.names.asName
@@ -17,11 +17,7 @@ import kotlin.reflect.typeOf
 /**
  * Reader of a custom object from input
  */
-public interface IOReader<out T> {
+public fun interface IOReader<out T> {
-    /**
-     * The type of object being read
-     */
-    public val type: KType
 
     public fun readFrom(source: Source): T
 
@@ -32,7 +28,6 @@ public interface IOReader<out T> {
          * no-op reader for binaries.
          */
         public val binary: IOReader<Binary> = object : IOReader<Binary> {
-            override val type: KType = typeOf<Binary>()
 
             override fun readFrom(source: Source): Binary = source.readByteArray().asBinary()
 
@@ -42,8 +37,6 @@ public interface IOReader<out T> {
 }
 
 public inline fun <reified T> IOReader(crossinline read: Source.() -> T): IOReader<T> = object : IOReader<T> {
-    override val type: KType = typeOf<T>()
-
     override fun readFrom(source: Source): T = source.read()
 }
 
@@ -56,24 +49,24 @@ public fun interface IOWriter<in T> {
  */
 public interface IOFormat<T> : IOReader<T>, IOWriter<T>
 
-public fun <T : Any> Source.readWith(format: IOReader<T>): T = format.readFrom(this)
+public fun <T> Source.readWith(format: IOReader<T>): T = format.readFrom(this)
 
 /**
  * Read given binary as an object using given format
  */
-public fun <T : Any> Binary.readWith(format: IOReader<T>): T = read {
+public fun <T> Binary.readWith(format: IOReader<T>): T = read {
-    readWith(format)
+    this.readWith(format)
 }
 
 /**
  * Write an object to the [Sink] with given [format]
  */
-public fun <T : Any> Sink.writeWith(format: IOWriter<T>, obj: T): Unit =
+public fun <T> Sink.writeWith(format: IOWriter<T>, obj: T): Unit =
     format.writeTo(this, obj)
 
 
-@DfId(IO_FORMAT_TYPE)
+@DfType(IO_FORMAT_TYPE)
-public interface IOFormatFactory<T : Any> : Factory<IOFormat<T>>, Named {
+public interface IOFormatFactory<T> : Factory<IOFormat<T>>, Named {
     /**
      * Explicit type for dynamic type checks
      */
@@ -86,7 +79,7 @@ public interface IOFormatFactory<T : Any> : Factory<IOFormat<T>>, Named {
     }
 }
 
-public fun <T : Any> Binary(obj: T, format: IOWriter<T>): Binary = Binary { format.writeTo(this, obj) }
+public fun <T> Binary(obj: T, format: IOWriter<T>): Binary = Binary { format.writeTo(this, obj) }
 
 public object FloatIOFormat : IOFormat<Float>, IOFormatFactory<Float> {
     override fun build(context: Context, meta: Meta): IOFormat<Float> = this

dataforge-io/src/commonMain/kotlin/space/kscience/dataforge/io/IOPlugin.kt

@@ -5,30 +5,12 @@ import space.kscience.dataforge.io.EnvelopeFormatFactory.Companion.ENVELOPE_FORM
 import space.kscience.dataforge.io.IOFormatFactory.Companion.IO_FORMAT_TYPE
 import space.kscience.dataforge.io.MetaFormatFactory.Companion.META_FORMAT_TYPE
 import space.kscience.dataforge.meta.Meta
-import space.kscience.dataforge.meta.get
 import space.kscience.dataforge.meta.string
-import space.kscience.dataforge.misc.DFInternal
 import space.kscience.dataforge.names.Name
-import kotlin.reflect.KType
-import kotlin.reflect.typeOf
 
 public class IOPlugin(meta: Meta) : AbstractPlugin(meta) {
     override val tag: PluginTag get() = Companion.tag
 
-    public val ioFormatFactories: Collection<IOFormatFactory<*>> by lazy {
-        context.gather<IOFormatFactory<*>>(IO_FORMAT_TYPE).values
-    }
-
-    @Suppress("UNCHECKED_CAST")
-    @DFInternal
-    public fun <T : Any> resolveIOFormat(type: KType, meta: Meta): IOFormat<T>? =
-        ioFormatFactories.singleOrNull { it.type == type }?.build(context, meta) as? IOFormat<T>
-
-    @OptIn(DFInternal::class)
-    public inline fun <reified T : Any> resolveIOFormat(meta: Meta = Meta.EMPTY): IOFormat<T>? =
-        resolveIOFormat(typeOf<T>(), meta)
-
-
     public val metaFormatFactories: Collection<MetaFormatFactory> by lazy {
         context.gather<MetaFormatFactory>(META_FORMAT_TYPE).values
     }

dataforge-io/src/commonMain/kotlin/space/kscience/dataforge/io/MetaFormat.kt

@@ -9,7 +9,7 @@ import space.kscience.dataforge.context.Global
 import space.kscience.dataforge.io.MetaFormatFactory.Companion.META_FORMAT_TYPE
 import space.kscience.dataforge.meta.Meta
 import space.kscience.dataforge.meta.descriptors.MetaDescriptor
-import space.kscience.dataforge.misc.DfId
+import space.kscience.dataforge.misc.DfType
 import space.kscience.dataforge.names.Name
 import space.kscience.dataforge.names.asName
 import space.kscience.dataforge.names.plus
@@ -21,8 +21,6 @@ import kotlin.reflect.typeOf
  */
 public interface MetaFormat : IOFormat<Meta> {
 
-    override val type: KType get() = typeOf<Meta>()
-
     override fun writeTo(sink: Sink, obj: Meta) {
         writeMeta(sink, obj, null)
     }
@@ -38,7 +36,7 @@ public interface MetaFormat : IOFormat<Meta> {
     public fun readMeta(source: Source, descriptor: MetaDescriptor? = null): Meta
 }
 
-@DfId(META_FORMAT_TYPE)
+@DfType(META_FORMAT_TYPE)
 public interface MetaFormatFactory : IOFormatFactory<Meta>, MetaFormat {
     public val shortName: String
 

dataforge-io/src/commonMain/kotlin/space/kscience/dataforge/io/Responder.kt

@@ -1,12 +0,0 @@
-package space.kscience.dataforge.io
-
-/**
- * An object that could respond to external messages asynchronously
- */
-public interface Responder {
-    /**
-     * Send a request and wait for response for this specific request
-     */
-    public suspend fun respond(request: Envelope): Envelope
-}
-

dataforge-io/src/commonMain/kotlin/space/kscience/dataforge/io/ioMisc.kt

@@ -5,7 +5,6 @@ import kotlinx.io.bytestring.ByteString
 import kotlinx.io.bytestring.decodeToString
 import kotlinx.io.bytestring.encodeToByteString
 import space.kscience.dataforge.meta.Meta
-import space.kscience.dataforge.misc.DFExperimental
 import kotlin.math.min
 
 /**
@@ -52,7 +51,6 @@ public fun IOPlugin.peekBinaryEnvelopeFormat(binary: Binary): EnvelopeFormat? {
 /**
  * A zero-copy read from
  */
-@DFExperimental
 public fun IOPlugin.readEnvelope(
     binary: Binary,
     readNonEnvelopes: Boolean = false,
@@ -62,7 +60,6 @@ public fun IOPlugin.readEnvelope(
     Envelope(Meta.EMPTY, binary)
 } else error("Can't infer format for $binary")
 
-@DFExperimental
 public fun IOPlugin.readEnvelope(
     string: String,
     readNonEnvelopes: Boolean = false,

dataforge-io/src/jvmMain/kotlin/space/kscience/dataforge/io/fileIO.kt

@@ -15,8 +15,6 @@ import java.nio.file.Path
 import java.nio.file.StandardOpenOption
 import kotlin.io.path.inputStream
 import kotlin.math.min
-import kotlin.reflect.full.isSupertypeOf
-import kotlin.reflect.typeOf
 import kotlin.streams.asSequence
 
 
@@ -77,18 +75,8 @@ public fun Path.rewrite(block: Sink.() -> Unit): Unit {
     stream.asSink().buffered().use(block)
 }
 
-@DFExperimental
 public fun EnvelopeFormat.readFile(path: Path): Envelope = readFrom(path.asBinary())
 
-/**
- * Resolve IOFormat based on type
- */
-@Suppress("UNCHECKED_CAST")
-@DFExperimental
-public inline fun <reified T : Any> IOPlugin.resolveIOFormat(): IOFormat<T>? =
-    ioFormatFactories.find { it.type.isSupertypeOf(typeOf<T>()) } as IOFormat<T>?
-
-
 public val IOPlugin.Companion.META_FILE_NAME: String get() = "@meta"
 public val IOPlugin.Companion.DATA_FILE_NAME: String get() = "@data"
 
@@ -192,7 +180,6 @@ public fun IOPlugin.peekFileEnvelopeFormat(path: Path): EnvelopeFormat? {
  *
  * Return null otherwise.
  */
-@DFExperimental
 public fun IOPlugin.readEnvelopeFile(
     path: Path,
     readNonEnvelopes: Boolean = false,

dataforge-meta/README.md

@@ -2,22 +2,27 @@
 
 Meta definition and basic operations on meta
 
+## Features
+
+ - [Meta](src/commonMain/kotlin/space/kscience/dataforge/meta/Meta.kt) : **Meta** is the representation of basic DataForge concept: Metadata, but it also could be called meta-value tree.
+ - [Value](src/commonMain/kotlin/space/kscience/dataforge/meta/Value.kt) : **Value** a sum type for different meta values.
+ - [Name](src/commonMain/kotlin/space/kscience/dataforge/names/Name.kt) : **Name** is an identifier to access tree-like structure.
+
+
 ## Usage
 
 ## Artifact:
 
-The Maven coordinates of this project are `space.kscience:dataforge-meta:0.7.0`.
+The Maven coordinates of this project are `space.kscience:dataforge-meta:0.10.0`.
 
 **Gradle Kotlin DSL:**
 ```kotlin
 repositories {
     maven("https://repo.kotlin.link")
-    //uncomment to access development builds
-    //maven("https://maven.pkg.jetbrains.space/spc/p/sci/dev")
     mavenCentral()
 }
 
 dependencies {
-    implementation("space.kscience:dataforge-meta:0.7.0")
+    implementation("space.kscience:dataforge-meta:0.10.0")
 }
 ```

dataforge-meta/api/dataforge-meta.api

@@ -1,3 +1,15 @@
+public final class space/kscience/dataforge/meta/ByteArrayValue : java/lang/Iterable, kotlin/jvm/internal/markers/KMappedMarker, space/kscience/dataforge/meta/Value {
+	public fun <init> ([B)V
+	public fun equals (Ljava/lang/Object;)Z
+	public fun getList ()Ljava/util/List;
+	public fun getType ()Lspace/kscience/dataforge/meta/ValueType;
+	public synthetic fun getValue ()Ljava/lang/Object;
+	public fun getValue ()[B
+	public fun hashCode ()I
+	public fun iterator ()Ljava/util/Iterator;
+	public fun toString ()Ljava/lang/String;
+}
+
 public abstract interface class space/kscience/dataforge/meta/Configurable {
 	public abstract fun getMeta ()Lspace/kscience/dataforge/meta/MutableMeta;
 }
@@ -30,7 +42,22 @@ public final class space/kscience/dataforge/meta/EnumValue : space/kscience/data
 }
 
 public final class space/kscience/dataforge/meta/ExoticValuesKt {
+	public static final fun asValue ([B)Lspace/kscience/dataforge/meta/Value;
 	public static final fun asValue ([D)Lspace/kscience/dataforge/meta/Value;
+	public static final fun byteArray (Lspace/kscience/dataforge/meta/MetaProvider;[BLspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadOnlyProperty;
+	public static final fun byteArray (Lspace/kscience/dataforge/meta/MutableMetaProvider;[BLspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun byteArray$default (Lspace/kscience/dataforge/meta/MetaProvider;[BLspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadOnlyProperty;
+	public static synthetic fun byteArray$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;[BLspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun doubleArray (Lspace/kscience/dataforge/meta/MetaProvider;[DLspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadOnlyProperty;
+	public static final fun doubleArray (Lspace/kscience/dataforge/meta/MutableMetaProvider;[DLspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun doubleArray$default (Lspace/kscience/dataforge/meta/MetaProvider;[DLspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadOnlyProperty;
+	public static synthetic fun doubleArray$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;[DLspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun getByteArray (Lspace/kscience/dataforge/meta/Meta;)[B
+	public static final fun getByteArray (Lspace/kscience/dataforge/meta/MetaConverter$Companion;)Lspace/kscience/dataforge/meta/MetaConverter;
+	public static final fun getByteArray (Lspace/kscience/dataforge/meta/Value;)[B
+	public static final fun getDoubleArray (Lspace/kscience/dataforge/meta/Meta;)[D
+	public static final fun getDoubleArray (Lspace/kscience/dataforge/meta/MetaConverter$Companion;)Lspace/kscience/dataforge/meta/MetaConverter;
+	public static final fun getDoubleArray (Lspace/kscience/dataforge/meta/Value;)[D
 	public static final fun lazyParseValue (Ljava/lang/String;)Lspace/kscience/dataforge/meta/LazyParsedValue;
 }
 
@@ -56,6 +83,20 @@ public final class space/kscience/dataforge/meta/JsonMetaKt {
 	public static final fun toValue (Lkotlinx/serialization/json/JsonPrimitive;Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;)Lspace/kscience/dataforge/meta/Value;
 }
 
+public final class space/kscience/dataforge/meta/KeepTransformationRule : space/kscience/dataforge/meta/TransformationRule {
+	public fun <init> (Lkotlin/jvm/functions/Function1;)V
+	public final fun component1 ()Lkotlin/jvm/functions/Function1;
+	public final fun copy (Lkotlin/jvm/functions/Function1;)Lspace/kscience/dataforge/meta/KeepTransformationRule;
+	public static synthetic fun copy$default (Lspace/kscience/dataforge/meta/KeepTransformationRule;Lkotlin/jvm/functions/Function1;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/KeepTransformationRule;
+	public fun equals (Ljava/lang/Object;)Z
+	public final fun getSelector ()Lkotlin/jvm/functions/Function1;
+	public fun hashCode ()I
+	public fun matches (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/Meta;)Z
+	public fun selectItems (Lspace/kscience/dataforge/meta/Meta;)Lkotlin/sequences/Sequence;
+	public fun toString ()Ljava/lang/String;
+	public fun transformItem (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/Meta;Lspace/kscience/dataforge/meta/MutableMeta;)V
+}
+
 public final class space/kscience/dataforge/meta/Laminate : space/kscience/dataforge/meta/TypedMeta {
 	public static final field Companion Lspace/kscience/dataforge/meta/Laminate$Companion;
 	public fun equals (Ljava/lang/Object;)Z
@@ -64,6 +105,8 @@ public final class space/kscience/dataforge/meta/Laminate : space/kscience/dataf
 	public synthetic fun get (Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/TypedMeta;
 	public fun getItems ()Ljava/util/Map;
 	public final fun getLayers ()Ljava/util/List;
+	public fun getSelf ()Lspace/kscience/dataforge/meta/Laminate;
+	public synthetic fun getSelf ()Lspace/kscience/dataforge/meta/TypedMeta;
 	public fun getValue ()Lspace/kscience/dataforge/meta/Value;
 	public fun hashCode ()I
 	public final fun merge ()Lspace/kscience/dataforge/meta/SealedMeta;
@@ -159,49 +202,89 @@ public final class space/kscience/dataforge/meta/MetaBuilder : space/kscience/da
 public abstract interface annotation class space/kscience/dataforge/meta/MetaBuilderMarker : java/lang/annotation/Annotation {
 }
 
+public abstract interface class space/kscience/dataforge/meta/MetaConverter : space/kscience/dataforge/meta/MetaReader {
+	public static final field Companion Lspace/kscience/dataforge/meta/MetaConverter$Companion;
+	public abstract fun convert (Ljava/lang/Object;)Lspace/kscience/dataforge/meta/Meta;
+	public fun getDescriptor ()Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;
+	public fun read (Lspace/kscience/dataforge/meta/Meta;)Ljava/lang/Object;
+	public abstract fun readOrNull (Lspace/kscience/dataforge/meta/Meta;)Ljava/lang/Object;
+}
+
+public final class space/kscience/dataforge/meta/MetaConverter$Companion {
+	public final fun getBoolean ()Lspace/kscience/dataforge/meta/MetaConverter;
+	public final fun getDouble ()Lspace/kscience/dataforge/meta/MetaConverter;
+	public final fun getFloat ()Lspace/kscience/dataforge/meta/MetaConverter;
+	public final fun getInt ()Lspace/kscience/dataforge/meta/MetaConverter;
+	public final fun getLong ()Lspace/kscience/dataforge/meta/MetaConverter;
+	public final fun getMeta ()Lspace/kscience/dataforge/meta/MetaConverter;
+	public final fun getNumber ()Lspace/kscience/dataforge/meta/MetaConverter;
+	public final fun getString ()Lspace/kscience/dataforge/meta/MetaConverter;
+	public final fun getStringList ()Lspace/kscience/dataforge/meta/MetaConverter;
+	public final fun getValue ()Lspace/kscience/dataforge/meta/MetaConverter;
+	public final fun valueList (Lkotlin/jvm/functions/Function1;Lkotlin/jvm/functions/Function1;)Lspace/kscience/dataforge/meta/MetaConverter;
+	public static synthetic fun valueList$default (Lspace/kscience/dataforge/meta/MetaConverter$Companion;Lkotlin/jvm/functions/Function1;Lkotlin/jvm/functions/Function1;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaConverter;
+}
+
+public final class space/kscience/dataforge/meta/MetaConverterKt {
+	public static final fun convertNullable (Lspace/kscience/dataforge/meta/MetaConverter;Ljava/lang/Object;)Lspace/kscience/dataforge/meta/Meta;
+}
+
+public abstract interface class space/kscience/dataforge/meta/MetaDelegate : kotlin/properties/ReadOnlyProperty, space/kscience/dataforge/meta/descriptors/Described {
+}
+
 public final class space/kscience/dataforge/meta/MetaDelegateKt {
-	public static final fun boolean (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadOnlyProperty;
+	public static final fun boolean (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static final fun boolean (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function0;)Lkotlin/properties/ReadOnlyProperty;
+	public static final fun boolean (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function0;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static final fun boolean (Lspace/kscience/dataforge/meta/MetaProvider;ZLspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadOnlyProperty;
+	public static final fun boolean (Lspace/kscience/dataforge/meta/MetaProvider;ZLspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static synthetic fun boolean$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadOnlyProperty;
+	public static synthetic fun boolean$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static synthetic fun boolean$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function0;ILjava/lang/Object;)Lkotlin/properties/ReadOnlyProperty;
+	public static synthetic fun boolean$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function0;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static synthetic fun boolean$default (Lspace/kscience/dataforge/meta/MetaProvider;ZLspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadOnlyProperty;
+	public static synthetic fun boolean$default (Lspace/kscience/dataforge/meta/MetaProvider;ZLspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static final fun double (Lspace/kscience/dataforge/meta/MetaProvider;DLspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadOnlyProperty;
+	public static final fun double (Lspace/kscience/dataforge/meta/MetaProvider;DLspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static final fun double (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadOnlyProperty;
+	public static final fun double (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static synthetic fun double$default (Lspace/kscience/dataforge/meta/MetaProvider;DLspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadOnlyProperty;
+	public static synthetic fun double$default (Lspace/kscience/dataforge/meta/MetaProvider;DLspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static synthetic fun double$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadOnlyProperty;
+	public static synthetic fun double$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static final fun float (Lspace/kscience/dataforge/meta/MetaProvider;FLspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadOnlyProperty;
+	public static final fun float (Lspace/kscience/dataforge/meta/MetaProvider;FLspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static final fun float (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadOnlyProperty;
+	public static final fun float (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static synthetic fun float$default (Lspace/kscience/dataforge/meta/MetaProvider;FLspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadOnlyProperty;
+	public static synthetic fun float$default (Lspace/kscience/dataforge/meta/MetaProvider;FLspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static synthetic fun float$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadOnlyProperty;
+	public static synthetic fun float$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static final fun int (Lspace/kscience/dataforge/meta/MetaProvider;ILspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadOnlyProperty;
+	public static final fun int (Lspace/kscience/dataforge/meta/MetaProvider;ILspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static final fun int (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadOnlyProperty;
+	public static final fun int (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static synthetic fun int$default (Lspace/kscience/dataforge/meta/MetaProvider;ILspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadOnlyProperty;
+	public static synthetic fun int$default (Lspace/kscience/dataforge/meta/MetaProvider;ILspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static synthetic fun int$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadOnlyProperty;
+	public static synthetic fun int$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static final fun long (Lspace/kscience/dataforge/meta/MetaProvider;JLspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadOnlyProperty;
+	public static final fun listOfReadable (Lspace/kscience/dataforge/meta/Meta;Lspace/kscience/dataforge/meta/MetaReader;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static final fun long (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadOnlyProperty;
+	public static synthetic fun listOfReadable$default (Lspace/kscience/dataforge/meta/Meta;Lspace/kscience/dataforge/meta/MetaReader;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static synthetic fun long$default (Lspace/kscience/dataforge/meta/MetaProvider;JLspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadOnlyProperty;
+	public static final fun listOfSpec (Lspace/kscience/dataforge/meta/Meta;Lspace/kscience/dataforge/meta/MetaReader;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static synthetic fun long$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadOnlyProperty;
+	public static synthetic fun listOfSpec$default (Lspace/kscience/dataforge/meta/Meta;Lspace/kscience/dataforge/meta/MetaReader;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static final fun node (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadOnlyProperty;
+	public static final fun long (Lspace/kscience/dataforge/meta/MetaProvider;JLspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static final fun node (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/transformations/MetaConverter;)Lkotlin/properties/ReadOnlyProperty;
+	public static final fun long (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static synthetic fun node$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadOnlyProperty;
+	public static synthetic fun long$default (Lspace/kscience/dataforge/meta/MetaProvider;JLspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static synthetic fun node$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/transformations/MetaConverter;ILjava/lang/Object;)Lkotlin/properties/ReadOnlyProperty;
+	public static synthetic fun long$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static final fun number (Lspace/kscience/dataforge/meta/MetaProvider;Ljava/lang/Number;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadOnlyProperty;
+	public static final fun node (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/MetaReader;)Lkotlin/properties/ReadOnlyProperty;
-	public static final fun number (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadOnlyProperty;
+	public static final fun node (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static final fun number (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function0;)Lkotlin/properties/ReadOnlyProperty;
+	public static synthetic fun node$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/MetaReader;ILjava/lang/Object;)Lkotlin/properties/ReadOnlyProperty;
-	public static synthetic fun number$default (Lspace/kscience/dataforge/meta/MetaProvider;Ljava/lang/Number;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadOnlyProperty;
+	public static synthetic fun node$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static synthetic fun number$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadOnlyProperty;
+	public static final fun number (Lspace/kscience/dataforge/meta/MetaProvider;Ljava/lang/Number;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static synthetic fun number$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function0;ILjava/lang/Object;)Lkotlin/properties/ReadOnlyProperty;
+	public static final fun number (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static final fun string (Lspace/kscience/dataforge/meta/MetaProvider;Ljava/lang/String;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadOnlyProperty;
+	public static final fun number (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function0;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static final fun string (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadOnlyProperty;
+	public static synthetic fun number$default (Lspace/kscience/dataforge/meta/MetaProvider;Ljava/lang/Number;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static final fun string (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function0;)Lkotlin/properties/ReadOnlyProperty;
+	public static synthetic fun number$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static synthetic fun string$default (Lspace/kscience/dataforge/meta/MetaProvider;Ljava/lang/String;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadOnlyProperty;
+	public static synthetic fun number$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function0;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static synthetic fun string$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadOnlyProperty;
+	public static final fun readable (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/meta/MetaReader;Ljava/lang/Object;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static synthetic fun string$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function0;ILjava/lang/Object;)Lkotlin/properties/ReadOnlyProperty;
+	public static final fun readable (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/meta/MetaReader;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static final fun value (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadOnlyProperty;
+	public static synthetic fun readable$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/meta/MetaReader;Ljava/lang/Object;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static final fun value (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function1;)Lkotlin/properties/ReadOnlyProperty;
+	public static synthetic fun readable$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/meta/MetaReader;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static synthetic fun value$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadOnlyProperty;
+	public static final fun spec (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/meta/MetaReader;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MetaDelegate;
-	public static synthetic fun value$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function1;ILjava/lang/Object;)Lkotlin/properties/ReadOnlyProperty;
+	public static synthetic fun spec$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/meta/MetaReader;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
+	public static final fun string (Lspace/kscience/dataforge/meta/MetaProvider;Ljava/lang/String;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MetaDelegate;
+	public static final fun string (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MetaDelegate;
+	public static final fun string (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function0;)Lspace/kscience/dataforge/meta/MetaDelegate;
+	public static synthetic fun string$default (Lspace/kscience/dataforge/meta/MetaProvider;Ljava/lang/String;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
+	public static synthetic fun string$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
+	public static synthetic fun string$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function0;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
+	public static final fun value (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;)Lspace/kscience/dataforge/meta/MetaDelegate;
+	public static final fun value (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;Lkotlin/jvm/functions/Function1;)Lspace/kscience/dataforge/meta/MetaDelegate;
+	public static synthetic fun value$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
+	public static synthetic fun value$default (Lspace/kscience/dataforge/meta/MetaProvider;Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;Lkotlin/jvm/functions/Function1;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MetaDelegate;
 }
 
 public final class space/kscience/dataforge/meta/MetaKt {
@@ -222,7 +305,6 @@ public final class space/kscience/dataforge/meta/MetaKt {
 	public static final fun getLong (Lspace/kscience/dataforge/meta/Meta;)Ljava/lang/Long;
 	public static final synthetic fun getNonNullable (Lspace/kscience/dataforge/meta/Meta;Lspace/kscience/dataforge/names/NameToken;)Lspace/kscience/dataforge/meta/Meta;
 	public static final fun getNumber (Lspace/kscience/dataforge/meta/Meta;)Ljava/lang/Number;
-	public static final fun getSelf (Lspace/kscience/dataforge/meta/TypedMeta;)Lspace/kscience/dataforge/meta/TypedMeta;
 	public static final fun getShort (Lspace/kscience/dataforge/meta/Meta;)Ljava/lang/Short;
 	public static final fun getString (Lspace/kscience/dataforge/meta/Meta;)Ljava/lang/String;
 	public static final fun getStringList (Lspace/kscience/dataforge/meta/Meta;)Ljava/util/List;
@@ -239,6 +321,20 @@ public abstract interface class space/kscience/dataforge/meta/MetaProvider : spa
 	public fun getValue (Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/Value;
 }
 
+public abstract interface class space/kscience/dataforge/meta/MetaReader : space/kscience/dataforge/meta/descriptors/Described {
+	public fun read (Lspace/kscience/dataforge/meta/Meta;)Ljava/lang/Object;
+	public abstract fun readOrNull (Lspace/kscience/dataforge/meta/Meta;)Ljava/lang/Object;
+}
+
+public final class space/kscience/dataforge/meta/MetaReaderKt {
+	public static final fun readNullable (Lspace/kscience/dataforge/meta/MetaReader;Lspace/kscience/dataforge/meta/Meta;)Ljava/lang/Object;
+	public static final fun readValue (Lspace/kscience/dataforge/meta/MetaReader;Lspace/kscience/dataforge/meta/Value;)Ljava/lang/Object;
+}
+
+public abstract interface class space/kscience/dataforge/meta/MetaRefStore : space/kscience/dataforge/meta/descriptors/Described {
+	public abstract fun getRefs ()Ljava/util/List;
+}
+
 public abstract interface class space/kscience/dataforge/meta/MetaRepr {
 	public abstract fun toMeta ()Lspace/kscience/dataforge/meta/Meta;
 }
@@ -252,6 +348,37 @@ public final class space/kscience/dataforge/meta/MetaSerializer : kotlinx/serial
 	public fun serialize (Lkotlinx/serialization/encoding/Encoder;Lspace/kscience/dataforge/meta/Meta;)V
 }
 
+public final class space/kscience/dataforge/meta/MetaTransformation {
+	public static final field Companion Lspace/kscience/dataforge/meta/MetaTransformation$Companion;
+	public static final fun apply-impl (Ljava/util/Collection;Lspace/kscience/dataforge/meta/Meta;)Lspace/kscience/dataforge/meta/Meta;
+	public static final fun bind-impl (Ljava/util/Collection;Lspace/kscience/dataforge/meta/ObservableMeta;Lspace/kscience/dataforge/meta/MutableMeta;)V
+	public static final synthetic fun box-impl (Ljava/util/Collection;)Lspace/kscience/dataforge/meta/MetaTransformation;
+	public static fun constructor-impl (Ljava/util/Collection;)Ljava/util/Collection;
+	public fun equals (Ljava/lang/Object;)Z
+	public static fun equals-impl (Ljava/util/Collection;Ljava/lang/Object;)Z
+	public static final fun equals-impl0 (Ljava/util/Collection;Ljava/util/Collection;)Z
+	public static final fun generate-impl (Ljava/util/Collection;Lspace/kscience/dataforge/meta/Meta;)Lspace/kscience/dataforge/meta/Meta;
+	public fun hashCode ()I
+	public static fun hashCode-impl (Ljava/util/Collection;)I
+	public fun toString ()Ljava/lang/String;
+	public static fun toString-impl (Ljava/util/Collection;)Ljava/lang/String;
+	public final synthetic fun unbox-impl ()Ljava/util/Collection;
+}
+
+public final class space/kscience/dataforge/meta/MetaTransformation$Companion {
+	public final fun make--mWxz5M (Lkotlin/jvm/functions/Function1;)Ljava/util/Collection;
+}
+
+public final class space/kscience/dataforge/meta/MetaTransformationBuilder {
+	public fun <init> ()V
+	public final fun build-m6Fha10 ()Ljava/util/Collection;
+	public final fun keep (Ljava/lang/String;)V
+	public final fun keep (Lkotlin/jvm/functions/Function1;)V
+	public final fun keep (Lspace/kscience/dataforge/names/Name;)V
+	public final fun move (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function1;)V
+	public static synthetic fun move$default (Lspace/kscience/dataforge/meta/MetaTransformationBuilder;Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function1;ILjava/lang/Object;)V
+}
+
 public abstract interface class space/kscience/dataforge/meta/MutableMeta : space/kscience/dataforge/meta/Meta, space/kscience/dataforge/meta/MutableMetaProvider {
 	public static final field Companion Lspace/kscience/dataforge/meta/MutableMeta$Companion;
 	public synthetic fun get (Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/Meta;
@@ -267,6 +394,7 @@ public abstract interface class space/kscience/dataforge/meta/MutableMeta : spac
 	public fun put (Ljava/lang/String;Lspace/kscience/dataforge/meta/MetaRepr;)V
 	public fun put (Ljava/lang/String;Lspace/kscience/dataforge/meta/Value;)V
 	public fun put (Ljava/lang/String;Z)V
+	public fun put (Ljava/lang/String;[B)V
 	public fun put (Ljava/lang/String;[D)V
 	public fun put (Lspace/kscience/dataforge/names/Name;Ljava/lang/Enum;)V
 	public fun put (Lspace/kscience/dataforge/names/Name;Ljava/lang/Number;)V
@@ -286,64 +414,70 @@ public final class space/kscience/dataforge/meta/MutableMeta$Companion {
 	public final fun serializer ()Lkotlinx/serialization/KSerializer;
 }
 
+public abstract interface class space/kscience/dataforge/meta/MutableMetaDelegate : kotlin/properties/ReadWriteProperty, space/kscience/dataforge/meta/descriptors/Described {
+}
+
 public final class space/kscience/dataforge/meta/MutableMetaDelegateKt {
-	public static final fun boolean (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun boolean (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun boolean (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function0;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun boolean (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function0;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun boolean (Lspace/kscience/dataforge/meta/MutableMetaProvider;ZLspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun boolean (Lspace/kscience/dataforge/meta/MutableMetaProvider;ZLspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun boolean$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun boolean$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun boolean$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function0;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun boolean$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function0;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun boolean$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;ZLspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun boolean$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;ZLspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun double (Lspace/kscience/dataforge/meta/MutableMetaProvider;DLspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun convertable (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/meta/MetaConverter;Ljava/lang/Object;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun double (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun convertable (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/meta/MetaConverter;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun double$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;DLspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun convertable$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/meta/MetaConverter;Ljava/lang/Object;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun double$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun convertable$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/meta/MetaConverter;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun doubleArray (Lspace/kscience/dataforge/meta/MutableMetaProvider;[DLspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun double (Lspace/kscience/dataforge/meta/MutableMetaProvider;DLspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun doubleArray$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;[DLspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun double (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun float (Lspace/kscience/dataforge/meta/MutableMetaProvider;FLspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun double$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;DLspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun float (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun double$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun float$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;FLspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun float (Lspace/kscience/dataforge/meta/MutableMetaProvider;FLspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun float$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun float (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun int (Lspace/kscience/dataforge/meta/MutableMetaProvider;ILspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun float$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;FLspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun int (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun float$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun int$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;ILspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun int (Lspace/kscience/dataforge/meta/MutableMetaProvider;ILspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun int$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun int (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun listValue (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function1;Lkotlin/jvm/functions/Function1;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun int$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;ILspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun listValue$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function1;Lkotlin/jvm/functions/Function1;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun int$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun long (Lspace/kscience/dataforge/meta/MutableMetaProvider;JLspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun listOfConvertable (Lspace/kscience/dataforge/meta/MutableMeta;Lspace/kscience/dataforge/meta/MetaConverter;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun long (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun listOfConvertable$default (Lspace/kscience/dataforge/meta/MutableMeta;Lspace/kscience/dataforge/meta/MetaConverter;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun long$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;JLspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun listValue (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function1;Lkotlin/jvm/functions/Function1;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun long$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun listValue$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function1;Lkotlin/jvm/functions/Function1;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun node (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun long (Lspace/kscience/dataforge/meta/MutableMetaProvider;JLspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun node (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/transformations/MetaConverter;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun long (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun node$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun long$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;JLspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun node$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/transformations/MetaConverter;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun long$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun number (Lspace/kscience/dataforge/meta/MutableMetaProvider;Ljava/lang/Number;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun node (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun number (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun node$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun number (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function0;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun number (Lspace/kscience/dataforge/meta/MutableMetaProvider;Ljava/lang/Number;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun number$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Ljava/lang/Number;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun number (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun number$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun number (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function0;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun number$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function0;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun number$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Ljava/lang/Number;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun numberList (Lspace/kscience/dataforge/meta/MutableMetaProvider;[Ljava/lang/Number;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun number$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun numberList$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;[Ljava/lang/Number;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun number$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function0;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun string (Lspace/kscience/dataforge/meta/MutableMetaProvider;Ljava/lang/String;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun numberList (Lspace/kscience/dataforge/meta/MutableMetaProvider;[Ljava/lang/Number;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun string (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun numberList$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;[Ljava/lang/Number;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun string (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function0;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun string (Lspace/kscience/dataforge/meta/MutableMetaProvider;Ljava/lang/String;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun string$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Ljava/lang/String;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun string (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun string$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun string (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function0;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun string$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function0;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun string$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Ljava/lang/String;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun stringList (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun string$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun stringList (Lspace/kscience/dataforge/meta/MutableMetaProvider;[Ljava/lang/String;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun string$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function0;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun stringList$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun stringList (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun stringList$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;[Ljava/lang/String;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun stringList (Lspace/kscience/dataforge/meta/MutableMetaProvider;[Ljava/lang/String;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun value (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun stringList$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static final fun value (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function1;Lkotlin/jvm/functions/Function1;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun stringList$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;[Ljava/lang/String;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun value$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun value (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function1;Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;Lkotlin/jvm/functions/Function1;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
-	public static synthetic fun value$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function1;Lkotlin/jvm/functions/Function1;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun value (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
+	public static synthetic fun value$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function1;Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;Lkotlin/jvm/functions/Function1;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
+	public static synthetic fun value$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/MutableMetaDelegate;
 }
 
 public final class space/kscience/dataforge/meta/MutableMetaKt {
 	public static final fun ObservableMutableMeta ()Lspace/kscience/dataforge/meta/ObservableMutableMeta;
 	public static final fun ObservableMutableMeta (Lkotlin/jvm/functions/Function1;)Lspace/kscience/dataforge/meta/ObservableMutableMeta;
+	public static final fun ObservableMutableMeta (Lspace/kscience/dataforge/meta/Meta;)Lspace/kscience/dataforge/meta/ObservableMutableMeta;
 	public static synthetic fun ObservableMutableMeta$default (Lkotlin/jvm/functions/Function1;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/ObservableMutableMeta;
 	public static final fun append (Lspace/kscience/dataforge/meta/MutableMeta;Ljava/lang/String;Lspace/kscience/dataforge/meta/Meta;)V
 	public static final fun append (Lspace/kscience/dataforge/meta/MutableMeta;Ljava/lang/String;Lspace/kscience/dataforge/meta/Value;)V
@@ -357,6 +491,7 @@ public final class space/kscience/dataforge/meta/MutableMetaKt {
 	public static final fun getOrCreate (Lspace/kscience/dataforge/meta/MutableTypedMeta;Ljava/lang/String;)Lspace/kscience/dataforge/meta/MutableTypedMeta;
 	public static final fun remove (Lspace/kscience/dataforge/meta/MutableMetaProvider;Ljava/lang/String;)V
 	public static final fun remove (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;)V
+	public static final fun reset (Lspace/kscience/dataforge/meta/MutableMeta;Lspace/kscience/dataforge/meta/Meta;)V
 	public static final fun set (Lspace/kscience/dataforge/meta/MutableMetaProvider;Ljava/lang/String;Ljava/lang/Iterable;)V
 	public static final fun set (Lspace/kscience/dataforge/meta/MutableMetaProvider;Ljava/lang/String;Lspace/kscience/dataforge/meta/Meta;)V
 	public static final fun set (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Ljava/lang/Iterable;)V
@@ -365,7 +500,7 @@ public final class space/kscience/dataforge/meta/MutableMetaKt {
 	public static final fun set (Lspace/kscience/dataforge/meta/MutableValueProvider;Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/Value;)V
 	public static final fun setIndexed (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Ljava/lang/Iterable;Lkotlin/jvm/functions/Function2;)V
 	public static synthetic fun setIndexed$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/names/Name;Ljava/lang/Iterable;Lkotlin/jvm/functions/Function2;ILjava/lang/Object;)V
-	public static final fun toMutableMeta (Lspace/kscience/dataforge/meta/Meta;)Lspace/kscience/dataforge/meta/ObservableMutableMeta;
+	public static final fun toMutableMeta (Lspace/kscience/dataforge/meta/Meta;)Lspace/kscience/dataforge/meta/MutableMeta;
 	public static final fun update (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/meta/Meta;)V
 	public static final fun withDefault (Lspace/kscience/dataforge/meta/MutableMeta;Lspace/kscience/dataforge/meta/MetaProvider;)Lspace/kscience/dataforge/meta/MutableMeta;
 }
@@ -385,8 +520,16 @@ public final class space/kscience/dataforge/meta/MutableMetaSerializer : kotlinx
 	public fun serialize (Lkotlinx/serialization/encoding/Encoder;Lspace/kscience/dataforge/meta/MutableMeta;)V
 }
 
+public final class space/kscience/dataforge/meta/MutableMetaViewKt {
+	public static final fun view (Lspace/kscience/dataforge/meta/MutableMeta;Ljava/lang/String;)Lspace/kscience/dataforge/meta/MutableMeta;
+	public static final fun view (Lspace/kscience/dataforge/meta/MutableMeta;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableMeta;
+}
+
 public abstract interface class space/kscience/dataforge/meta/MutableTypedMeta : space/kscience/dataforge/meta/MutableMeta, space/kscience/dataforge/meta/TypedMeta {
-	public abstract fun get (Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableTypedMeta;
+	public synthetic fun get (Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/Meta;
+	public synthetic fun get (Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableMeta;
+	public fun get (Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableTypedMeta;
+	public synthetic fun get (Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/TypedMeta;
 	public abstract fun getOrCreate (Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableTypedMeta;
 }
 
@@ -419,32 +562,48 @@ public abstract interface class space/kscience/dataforge/meta/ObservableMeta : s
 	public abstract fun removeListener (Ljava/lang/Object;)V
 }
 
-public final class space/kscience/dataforge/meta/ObservableMetaKt {
-	public static final fun useProperty (Lspace/kscience/dataforge/meta/Scheme;Lkotlin/reflect/KProperty1;Ljava/lang/Object;Lkotlin/jvm/functions/Function2;)V
-	public static synthetic fun useProperty$default (Lspace/kscience/dataforge/meta/Scheme;Lkotlin/reflect/KProperty1;Ljava/lang/Object;Lkotlin/jvm/functions/Function2;ILjava/lang/Object;)V
-}
-
 public final class space/kscience/dataforge/meta/ObservableMetaWrapperKt {
 	public static final fun asObservable (Lspace/kscience/dataforge/meta/MutableMeta;)Lspace/kscience/dataforge/meta/ObservableMutableMeta;
 }
 
 public abstract interface class space/kscience/dataforge/meta/ObservableMutableMeta : space/kscience/dataforge/meta/MutableMeta, space/kscience/dataforge/meta/MutableTypedMeta, space/kscience/dataforge/meta/ObservableMeta {
-	public synthetic fun get (Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/Meta;
+	public static final field Companion Lspace/kscience/dataforge/meta/ObservableMutableMeta$Companion;
-	public synthetic fun get (Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableMeta;
+	public fun getSelf ()Lspace/kscience/dataforge/meta/ObservableMutableMeta;
-	public synthetic fun get (Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableTypedMeta;
+	public synthetic fun getSelf ()Lspace/kscience/dataforge/meta/TypedMeta;
-	public fun get (Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/ObservableMutableMeta;
-	public synthetic fun get (Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/TypedMeta;
-	public abstract fun getOrCreate (Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/ObservableMutableMeta;
 }
 
-public abstract interface class space/kscience/dataforge/meta/ReadOnlySpecification : space/kscience/dataforge/meta/descriptors/Described {
+public final class space/kscience/dataforge/meta/ObservableMutableMeta$Companion {
-	public abstract fun empty ()Ljava/lang/Object;
+	public final fun serializer ()Lkotlinx/serialization/KSerializer;
-	public fun invoke (Lkotlin/jvm/functions/Function1;)Ljava/lang/Object;
+}
-	public abstract fun read (Lspace/kscience/dataforge/meta/Meta;)Ljava/lang/Object;
+
+public final class space/kscience/dataforge/meta/ObservableMutableMetaSerializer : kotlinx/serialization/KSerializer {
+	public static final field INSTANCE Lspace/kscience/dataforge/meta/ObservableMutableMetaSerializer;
+	public synthetic fun deserialize (Lkotlinx/serialization/encoding/Decoder;)Ljava/lang/Object;
+	public fun deserialize (Lkotlinx/serialization/encoding/Decoder;)Lspace/kscience/dataforge/meta/ObservableMutableMeta;
+	public fun getDescriptor ()Lkotlinx/serialization/descriptors/SerialDescriptor;
+	public synthetic fun serialize (Lkotlinx/serialization/encoding/Encoder;Ljava/lang/Object;)V
+	public fun serialize (Lkotlinx/serialization/encoding/Encoder;Lspace/kscience/dataforge/meta/ObservableMutableMeta;)V
+}
+
+public final class space/kscience/dataforge/meta/RegexItemTransformationRule : space/kscience/dataforge/meta/TransformationRule {
+	public fun <init> (Lkotlin/text/Regex;Lkotlin/jvm/functions/Function4;)V
+	public final fun component1 ()Lkotlin/text/Regex;
+	public final fun component2 ()Lkotlin/jvm/functions/Function4;
+	public final fun copy (Lkotlin/text/Regex;Lkotlin/jvm/functions/Function4;)Lspace/kscience/dataforge/meta/RegexItemTransformationRule;
+	public static synthetic fun copy$default (Lspace/kscience/dataforge/meta/RegexItemTransformationRule;Lkotlin/text/Regex;Lkotlin/jvm/functions/Function4;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/RegexItemTransformationRule;
+	public fun equals (Ljava/lang/Object;)Z
+	public final fun getFrom ()Lkotlin/text/Regex;
+	public final fun getTransform ()Lkotlin/jvm/functions/Function4;
+	public fun hashCode ()I
+	public fun matches (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/Meta;)Z
+	public fun toString ()Ljava/lang/String;
+	public fun transformItem (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/Meta;Lspace/kscience/dataforge/meta/MutableMeta;)V
 }
 
 public class space/kscience/dataforge/meta/Scheme : space/kscience/dataforge/meta/Configurable, space/kscience/dataforge/meta/MetaRepr, space/kscience/dataforge/meta/MutableMetaProvider, space/kscience/dataforge/meta/descriptors/Described {
 	public fun <init> ()V
+	public fun <init> (Lspace/kscience/dataforge/meta/Meta;Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;)V
+	public synthetic fun <init> (Lspace/kscience/dataforge/meta/Meta;Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;ILkotlin/jvm/internal/DefaultConstructorMarker;)V
 	public synthetic fun get (Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/Meta;
 	public fun get (Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/MutableMeta;
 	public final fun getDescriptor ()Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;
@@ -454,6 +613,7 @@ public class space/kscience/dataforge/meta/Scheme : space/kscience/dataforge/met
 	public fun setValue (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/Value;)V
 	public fun toMeta ()Lspace/kscience/dataforge/meta/Laminate;
 	public synthetic fun toMeta ()Lspace/kscience/dataforge/meta/Meta;
+	public fun toString ()Ljava/lang/String;
 	public fun validate (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/Meta;)Z
 }
 
@@ -461,20 +621,34 @@ public final class space/kscience/dataforge/meta/SchemeKt {
 	public static final fun copy (Lspace/kscience/dataforge/meta/Scheme;Lspace/kscience/dataforge/meta/SchemeSpec;Lkotlin/jvm/functions/Function1;)Lspace/kscience/dataforge/meta/Scheme;
 	public static synthetic fun copy$default (Lspace/kscience/dataforge/meta/Scheme;Lspace/kscience/dataforge/meta/SchemeSpec;Lkotlin/jvm/functions/Function1;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/Scheme;
 	public static final fun invoke (Lspace/kscience/dataforge/meta/Scheme;Lkotlin/jvm/functions/Function1;)Lspace/kscience/dataforge/meta/Scheme;
-	public static final fun retarget (Lspace/kscience/dataforge/meta/Scheme;Lspace/kscience/dataforge/meta/MutableMeta;)Lspace/kscience/dataforge/meta/Scheme;
+	public static final fun listOfScheme (Lspace/kscience/dataforge/meta/MutableMeta;Lspace/kscience/dataforge/meta/SchemeSpec;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun listOfScheme (Lspace/kscience/dataforge/meta/Scheme;Lspace/kscience/dataforge/meta/SchemeSpec;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun listOfScheme$default (Lspace/kscience/dataforge/meta/MutableMeta;Lspace/kscience/dataforge/meta/SchemeSpec;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun listOfScheme$default (Lspace/kscience/dataforge/meta/Scheme;Lspace/kscience/dataforge/meta/SchemeSpec;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun scheme (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/meta/SchemeSpec;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun scheme (Lspace/kscience/dataforge/meta/Scheme;Lspace/kscience/dataforge/meta/SchemeSpec;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun scheme$default (Lspace/kscience/dataforge/meta/MutableMetaProvider;Lspace/kscience/dataforge/meta/SchemeSpec;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun scheme$default (Lspace/kscience/dataforge/meta/Scheme;Lspace/kscience/dataforge/meta/SchemeSpec;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun schemeOrNull (Lspace/kscience/dataforge/meta/MutableMeta;Lspace/kscience/dataforge/meta/SchemeSpec;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun schemeOrNull (Lspace/kscience/dataforge/meta/Scheme;Lspace/kscience/dataforge/meta/SchemeSpec;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun schemeOrNull$default (Lspace/kscience/dataforge/meta/MutableMeta;Lspace/kscience/dataforge/meta/SchemeSpec;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun schemeOrNull$default (Lspace/kscience/dataforge/meta/Scheme;Lspace/kscience/dataforge/meta/SchemeSpec;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public static final fun updateWith (Lspace/kscience/dataforge/meta/Configurable;Lspace/kscience/dataforge/meta/SchemeSpec;Lkotlin/jvm/functions/Function1;)Lspace/kscience/dataforge/meta/Scheme;
+	public static final fun updateWith (Lspace/kscience/dataforge/meta/MutableMeta;Lspace/kscience/dataforge/meta/SchemeSpec;Lkotlin/jvm/functions/Function1;)Lspace/kscience/dataforge/meta/Scheme;
+	public static final fun useProperty (Lspace/kscience/dataforge/meta/Scheme;Lkotlin/reflect/KProperty1;Ljava/lang/Object;Lkotlin/jvm/functions/Function2;)V
+	public static synthetic fun useProperty$default (Lspace/kscience/dataforge/meta/Scheme;Lkotlin/reflect/KProperty1;Ljava/lang/Object;Lkotlin/jvm/functions/Function2;ILjava/lang/Object;)V
 }
 
-public class space/kscience/dataforge/meta/SchemeSpec : space/kscience/dataforge/meta/Specification {
+public class space/kscience/dataforge/meta/SchemeSpec : space/kscience/dataforge/meta/MetaConverter {
 	public fun <init> (Lkotlin/jvm/functions/Function0;)V
-	public synthetic fun empty ()Ljava/lang/Object;
+	public synthetic fun convert (Ljava/lang/Object;)Lspace/kscience/dataforge/meta/Meta;
-	public fun empty ()Lspace/kscience/dataforge/meta/Scheme;
+	public fun convert (Lspace/kscience/dataforge/meta/Scheme;)Lspace/kscience/dataforge/meta/Meta;
+	public final fun empty ()Lspace/kscience/dataforge/meta/Scheme;
 	public fun getDescriptor ()Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;
-	public synthetic fun invoke (Lkotlin/jvm/functions/Function1;)Ljava/lang/Object;
 	public final fun invoke (Lkotlin/jvm/functions/Function1;)Lspace/kscience/dataforge/meta/Scheme;
-	public synthetic fun read (Lspace/kscience/dataforge/meta/Meta;)Ljava/lang/Object;
+	public synthetic fun readOrNull (Lspace/kscience/dataforge/meta/Meta;)Ljava/lang/Object;
-	public fun read (Lspace/kscience/dataforge/meta/Meta;)Lspace/kscience/dataforge/meta/Scheme;
+	public fun readOrNull (Lspace/kscience/dataforge/meta/Meta;)Lspace/kscience/dataforge/meta/Scheme;
-	public synthetic fun write (Lspace/kscience/dataforge/meta/MutableMeta;)Ljava/lang/Object;
+	public final fun write (Lspace/kscience/dataforge/meta/MutableMeta;)Lspace/kscience/dataforge/meta/Scheme;
-	public fun write (Lspace/kscience/dataforge/meta/MutableMeta;)Lspace/kscience/dataforge/meta/Scheme;
 }
 
 public final class space/kscience/dataforge/meta/SealedMeta : space/kscience/dataforge/meta/TypedMeta {
@@ -482,19 +656,21 @@ public final class space/kscience/dataforge/meta/SealedMeta : space/kscience/dat
 	public fun <init> (Lspace/kscience/dataforge/meta/Value;Ljava/util/Map;)V
 	public fun equals (Ljava/lang/Object;)Z
 	public fun getItems ()Ljava/util/Map;
+	public fun getSelf ()Lspace/kscience/dataforge/meta/SealedMeta;
+	public synthetic fun getSelf ()Lspace/kscience/dataforge/meta/TypedMeta;
 	public fun getValue ()Lspace/kscience/dataforge/meta/Value;
 	public fun hashCode ()I
 	public fun toString ()Ljava/lang/String;
 }
 
-public final class space/kscience/dataforge/meta/SealedMeta$$serializer : kotlinx/serialization/internal/GeneratedSerializer {
+public synthetic class space/kscience/dataforge/meta/SealedMeta$$serializer : kotlinx/serialization/internal/GeneratedSerializer {
 	public static final field INSTANCE Lspace/kscience/dataforge/meta/SealedMeta$$serializer;
-	public fun childSerializers ()[Lkotlinx/serialization/KSerializer;
+	public final fun childSerializers ()[Lkotlinx/serialization/KSerializer;
 	public synthetic fun deserialize (Lkotlinx/serialization/encoding/Decoder;)Ljava/lang/Object;
-	public fun deserialize (Lkotlinx/serialization/encoding/Decoder;)Lspace/kscience/dataforge/meta/SealedMeta;
+	public final fun deserialize (Lkotlinx/serialization/encoding/Decoder;)Lspace/kscience/dataforge/meta/SealedMeta;
-	public fun getDescriptor ()Lkotlinx/serialization/descriptors/SerialDescriptor;
+	public final fun getDescriptor ()Lkotlinx/serialization/descriptors/SerialDescriptor;
 	public synthetic fun serialize (Lkotlinx/serialization/encoding/Encoder;Ljava/lang/Object;)V
-	public fun serialize (Lkotlinx/serialization/encoding/Encoder;Lspace/kscience/dataforge/meta/SealedMeta;)V
+	public final fun serialize (Lkotlinx/serialization/encoding/Encoder;Lspace/kscience/dataforge/meta/SealedMeta;)V
 	public fun typeParametersSerializers ()[Lkotlinx/serialization/KSerializer;
 }
 
@@ -515,21 +691,20 @@ public final class space/kscience/dataforge/meta/SealedMetaKt {
 	public static final fun seal (Lspace/kscience/dataforge/meta/Meta;)Lspace/kscience/dataforge/meta/SealedMeta;
 }
 
-public abstract interface class space/kscience/dataforge/meta/Specification : space/kscience/dataforge/meta/ReadOnlySpecification {
+public final class space/kscience/dataforge/meta/SingleItemTransformationRule : space/kscience/dataforge/meta/TransformationRule {
-	public abstract fun write (Lspace/kscience/dataforge/meta/MutableMeta;)Ljava/lang/Object;
+	public fun <init> (Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function3;)V
-}
+	public final fun component1 ()Lspace/kscience/dataforge/names/Name;
-
+	public final fun component2 ()Lkotlin/jvm/functions/Function3;
-public final class space/kscience/dataforge/meta/SpecificationKt {
+	public final fun copy (Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function3;)Lspace/kscience/dataforge/meta/SingleItemTransformationRule;
-	public static final fun spec (Lspace/kscience/dataforge/meta/MutableMeta;Lspace/kscience/dataforge/meta/Specification;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public static synthetic fun copy$default (Lspace/kscience/dataforge/meta/SingleItemTransformationRule;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function3;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/SingleItemTransformationRule;
-	public static final fun spec (Lspace/kscience/dataforge/meta/Scheme;Lspace/kscience/dataforge/meta/Specification;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public fun equals (Ljava/lang/Object;)Z
-	public static synthetic fun spec$default (Lspace/kscience/dataforge/meta/MutableMeta;Lspace/kscience/dataforge/meta/Specification;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public final fun getFrom ()Lspace/kscience/dataforge/names/Name;
-	public static synthetic fun spec$default (Lspace/kscience/dataforge/meta/Scheme;Lspace/kscience/dataforge/meta/Specification;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public final fun getTransform ()Lkotlin/jvm/functions/Function3;
-	public static final fun specOrNull (Lspace/kscience/dataforge/meta/MutableMeta;Lspace/kscience/dataforge/meta/Specification;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public fun hashCode ()I
-	public static final fun specOrNull (Lspace/kscience/dataforge/meta/Scheme;Lspace/kscience/dataforge/meta/Specification;Lspace/kscience/dataforge/names/Name;)Lkotlin/properties/ReadWriteProperty;
+	public fun matches (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/Meta;)Z
-	public static synthetic fun specOrNull$default (Lspace/kscience/dataforge/meta/MutableMeta;Lspace/kscience/dataforge/meta/Specification;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public fun selectItems (Lspace/kscience/dataforge/meta/Meta;)Lkotlin/sequences/Sequence;
-	public static synthetic fun specOrNull$default (Lspace/kscience/dataforge/meta/Scheme;Lspace/kscience/dataforge/meta/Specification;Lspace/kscience/dataforge/names/Name;ILjava/lang/Object;)Lkotlin/properties/ReadWriteProperty;
+	public fun toString ()Ljava/lang/String;
-	public static final fun updateWith (Lspace/kscience/dataforge/meta/Configurable;Lspace/kscience/dataforge/meta/Specification;Lkotlin/jvm/functions/Function1;)Ljava/lang/Object;
+	public fun transformItem (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/Meta;Lspace/kscience/dataforge/meta/MutableMeta;)V
-	public static final fun updateWith (Lspace/kscience/dataforge/meta/MutableMeta;Lspace/kscience/dataforge/meta/Specification;Lkotlin/jvm/functions/Function1;)Ljava/lang/Object;
 }
 
 public final class space/kscience/dataforge/meta/StringValue : space/kscience/dataforge/meta/Value {
@@ -550,6 +725,12 @@ public final class space/kscience/dataforge/meta/StringValue : space/kscience/da
 	public final synthetic fun unbox-impl ()Ljava/lang/String;
 }
 
+public abstract interface class space/kscience/dataforge/meta/TransformationRule {
+	public abstract fun matches (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/Meta;)Z
+	public fun selectItems (Lspace/kscience/dataforge/meta/Meta;)Lkotlin/sequences/Sequence;
+	public abstract fun transformItem (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/Meta;Lspace/kscience/dataforge/meta/MutableMeta;)V
+}
+
 public final class space/kscience/dataforge/meta/True : space/kscience/dataforge/meta/Value {
 	public static final field INSTANCE Lspace/kscience/dataforge/meta/True;
 	public fun equals (Ljava/lang/Object;)Z
@@ -563,6 +744,7 @@ public abstract interface class space/kscience/dataforge/meta/TypedMeta : space/
 	public synthetic fun get (Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/Meta;
 	public fun get (Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/meta/TypedMeta;
 	public abstract fun getItems ()Ljava/util/Map;
+	public abstract fun getSelf ()Lspace/kscience/dataforge/meta/TypedMeta;
 	public fun toMeta ()Lspace/kscience/dataforge/meta/Meta;
 }
 
@@ -587,7 +769,6 @@ public final class space/kscience/dataforge/meta/Value$Companion {
 public final class space/kscience/dataforge/meta/ValueExtensionsKt {
 	public static final fun getBoolean (Lspace/kscience/dataforge/meta/Value;)Z
 	public static final fun getDouble (Lspace/kscience/dataforge/meta/Value;)D
-	public static final fun getDoubleArray (Lspace/kscience/dataforge/meta/Value;)[D
 	public static final fun getFloat (Lspace/kscience/dataforge/meta/Value;)F
 	public static final fun getInt (Lspace/kscience/dataforge/meta/Value;)I
 	public static final fun getLong (Lspace/kscience/dataforge/meta/Value;)J
@@ -606,7 +787,6 @@ public final class space/kscience/dataforge/meta/ValueKt {
 	public static final fun asValue (Ljava/lang/Number;)Lspace/kscience/dataforge/meta/Value;
 	public static final fun asValue (Ljava/lang/String;)Lspace/kscience/dataforge/meta/Value;
 	public static final fun asValue (Z)Lspace/kscience/dataforge/meta/Value;
-	public static final fun asValue ([B)Lspace/kscience/dataforge/meta/Value;
 	public static final fun asValue ([F)Lspace/kscience/dataforge/meta/Value;
 	public static final fun asValue ([I)Lspace/kscience/dataforge/meta/Value;
 	public static final fun asValue ([J)Lspace/kscience/dataforge/meta/Value;
@@ -692,20 +872,21 @@ public final class space/kscience/dataforge/meta/descriptors/MetaDescriptor {
 	public final fun getDescription ()Ljava/lang/String;
 	public final fun getIndexKey ()Ljava/lang/String;
 	public final fun getMultiple ()Z
+	public final fun getNodes ()Ljava/util/Map;
 	public final fun getValueRestriction ()Lspace/kscience/dataforge/meta/descriptors/ValueRestriction;
 	public final fun getValueTypes ()Ljava/util/List;
 	public fun hashCode ()I
 	public fun toString ()Ljava/lang/String;
 }
 
-public final class space/kscience/dataforge/meta/descriptors/MetaDescriptor$$serializer : kotlinx/serialization/internal/GeneratedSerializer {
+public synthetic class space/kscience/dataforge/meta/descriptors/MetaDescriptor$$serializer : kotlinx/serialization/internal/GeneratedSerializer {
 	public static final field INSTANCE Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor$$serializer;
-	public fun childSerializers ()[Lkotlinx/serialization/KSerializer;
+	public final fun childSerializers ()[Lkotlinx/serialization/KSerializer;
 	public synthetic fun deserialize (Lkotlinx/serialization/encoding/Decoder;)Ljava/lang/Object;
-	public fun deserialize (Lkotlinx/serialization/encoding/Decoder;)Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;
+	public final fun deserialize (Lkotlinx/serialization/encoding/Decoder;)Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;
-	public fun getDescriptor ()Lkotlinx/serialization/descriptors/SerialDescriptor;
+	public final fun getDescriptor ()Lkotlinx/serialization/descriptors/SerialDescriptor;
 	public synthetic fun serialize (Lkotlinx/serialization/encoding/Encoder;Ljava/lang/Object;)V
-	public fun serialize (Lkotlinx/serialization/encoding/Encoder;Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;)V
+	public final fun serialize (Lkotlinx/serialization/encoding/Encoder;Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;)V
 	public fun typeParametersSerializers ()[Lkotlinx/serialization/KSerializer;
 }
 
@@ -720,25 +901,23 @@ public final class space/kscience/dataforge/meta/descriptors/MetaDescriptorBuild
 	public final fun attributes (Lkotlin/jvm/functions/Function1;)V
 	public final fun build ()Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;
 	public final fun default (Ljava/lang/Object;)V
+	public final fun from (Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;)V
 	public final fun getAllowedValues ()Ljava/util/List;
 	public final fun getAttributes ()Lspace/kscience/dataforge/meta/MutableMeta;
 	public final fun getChildren ()Ljava/util/Map;
 	public final fun getDefault ()Lspace/kscience/dataforge/meta/Value;
+	public final fun getDescription ()Ljava/lang/String;
 	public final fun getIndexKey ()Ljava/lang/String;
-	public final fun getInfo ()Ljava/lang/String;
 	public final fun getMultiple ()Z
 	public final fun getValueRestriction ()Lspace/kscience/dataforge/meta/descriptors/ValueRestriction;
 	public final fun getValueTypes ()Ljava/util/List;
-	public final fun item (Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function1;)Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;
+	public final fun node (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;)V
-	public static synthetic fun item$default (Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function1;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;
-	public final fun node (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;Lkotlin/jvm/functions/Function1;)Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;
-	public static synthetic fun node$default (Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;Lkotlin/jvm/functions/Function1;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;
 	public final fun setAllowedValues (Ljava/util/List;)V
 	public final fun setAttributes (Lspace/kscience/dataforge/meta/MutableMeta;)V
 	public final fun setChildren (Ljava/util/Map;)V
 	public final fun setDefault (Lspace/kscience/dataforge/meta/Value;)V
+	public final fun setDescription (Ljava/lang/String;)V
 	public final fun setIndexKey (Ljava/lang/String;)V
-	public final fun setInfo (Ljava/lang/String;)V
 	public final fun setMultiple (Z)V
 	public final fun setValueRestriction (Lspace/kscience/dataforge/meta/descriptors/ValueRestriction;)V
 	public final fun setValueTypes (Ljava/util/List;)V
@@ -749,16 +928,16 @@ public final class space/kscience/dataforge/meta/descriptors/MetaDescriptorBuild
 	public static final fun MetaDescriptor (Lkotlin/jvm/functions/Function1;)Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;
 	public static final fun copy (Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;Lkotlin/jvm/functions/Function1;)Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;
 	public static synthetic fun copy$default (Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;Lkotlin/jvm/functions/Function1;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;
-	public static final fun item (Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;Ljava/lang/String;Lkotlin/jvm/functions/Function1;)Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;
 	public static final fun node (Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;Ljava/lang/String;Lkotlin/jvm/functions/Function1;)V
 	public static final fun node (Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;Ljava/lang/String;Lspace/kscience/dataforge/meta/descriptors/Described;Lkotlin/jvm/functions/Function1;)V
-	public static final fun node (Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function1;)Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;
+	public static final fun node (Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;Ljava/lang/String;Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;)V
+	public static final fun node (Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function1;)V
 	public static synthetic fun node$default (Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;Ljava/lang/String;Lspace/kscience/dataforge/meta/descriptors/Described;Lkotlin/jvm/functions/Function1;ILjava/lang/Object;)V
 	public static final fun required (Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;)V
-	public static final fun value (Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;Ljava/lang/String;Lspace/kscience/dataforge/meta/ValueType;[Lspace/kscience/dataforge/meta/ValueType;Lkotlin/jvm/functions/Function1;)Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;
+	public static final fun value (Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;Ljava/lang/String;Lspace/kscience/dataforge/meta/ValueType;[Lspace/kscience/dataforge/meta/ValueType;Lkotlin/jvm/functions/Function1;)V
-	public static final fun value (Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/ValueType;[Lspace/kscience/dataforge/meta/ValueType;Lkotlin/jvm/functions/Function1;)Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;
+	public static final fun value (Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/ValueType;[Lspace/kscience/dataforge/meta/ValueType;Lkotlin/jvm/functions/Function1;)V
-	public static synthetic fun value$default (Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;Ljava/lang/String;Lspace/kscience/dataforge/meta/ValueType;[Lspace/kscience/dataforge/meta/ValueType;Lkotlin/jvm/functions/Function1;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;
+	public static synthetic fun value$default (Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;Ljava/lang/String;Lspace/kscience/dataforge/meta/ValueType;[Lspace/kscience/dataforge/meta/ValueType;Lkotlin/jvm/functions/Function1;ILjava/lang/Object;)V
-	public static synthetic fun value$default (Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/ValueType;[Lspace/kscience/dataforge/meta/ValueType;Lkotlin/jvm/functions/Function1;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;
+	public static synthetic fun value$default (Lspace/kscience/dataforge/meta/descriptors/MetaDescriptorBuilder;Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/ValueType;[Lspace/kscience/dataforge/meta/ValueType;Lkotlin/jvm/functions/Function1;ILjava/lang/Object;)V
 }
 
 public final class space/kscience/dataforge/meta/descriptors/MetaDescriptorKt {
@@ -772,6 +951,7 @@ public final class space/kscience/dataforge/meta/descriptors/MetaDescriptorKt {
 
 public final class space/kscience/dataforge/meta/descriptors/ValueRestriction : java/lang/Enum {
 	public static final field ABSENT Lspace/kscience/dataforge/meta/descriptors/ValueRestriction;
+	public static final field Companion Lspace/kscience/dataforge/meta/descriptors/ValueRestriction$Companion;
 	public static final field NONE Lspace/kscience/dataforge/meta/descriptors/ValueRestriction;
 	public static final field REQUIRED Lspace/kscience/dataforge/meta/descriptors/ValueRestriction;
 	public static fun getEntries ()Lkotlin/enums/EnumEntries;
@@ -779,119 +959,8 @@ public final class space/kscience/dataforge/meta/descriptors/ValueRestriction :
 	public static fun values ()[Lspace/kscience/dataforge/meta/descriptors/ValueRestriction;
 }
 
-public final class space/kscience/dataforge/meta/transformations/KeepTransformationRule : space/kscience/dataforge/meta/transformations/TransformationRule {
+public final class space/kscience/dataforge/meta/descriptors/ValueRestriction$Companion {
-	public fun <init> (Lkotlin/jvm/functions/Function1;)V
+	public final fun serializer ()Lkotlinx/serialization/KSerializer;
-	public final fun component1 ()Lkotlin/jvm/functions/Function1;
-	public final fun copy (Lkotlin/jvm/functions/Function1;)Lspace/kscience/dataforge/meta/transformations/KeepTransformationRule;
-	public static synthetic fun copy$default (Lspace/kscience/dataforge/meta/transformations/KeepTransformationRule;Lkotlin/jvm/functions/Function1;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/transformations/KeepTransformationRule;
-	public fun equals (Ljava/lang/Object;)Z
-	public final fun getSelector ()Lkotlin/jvm/functions/Function1;
-	public fun hashCode ()I
-	public fun matches (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/Meta;)Z
-	public fun selectItems (Lspace/kscience/dataforge/meta/Meta;)Lkotlin/sequences/Sequence;
-	public fun toString ()Ljava/lang/String;
-	public fun transformItem (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/Meta;Lspace/kscience/dataforge/meta/MutableMeta;)V
-}
-
-public abstract interface class space/kscience/dataforge/meta/transformations/MetaConverter {
-	public static final field Companion Lspace/kscience/dataforge/meta/transformations/MetaConverter$Companion;
-	public fun getDescriptor ()Lspace/kscience/dataforge/meta/descriptors/MetaDescriptor;
-	public abstract fun getType ()Lkotlin/reflect/KType;
-	public fun metaToObject (Lspace/kscience/dataforge/meta/Meta;)Ljava/lang/Object;
-	public abstract fun metaToObjectOrNull (Lspace/kscience/dataforge/meta/Meta;)Ljava/lang/Object;
-	public abstract fun objectToMeta (Ljava/lang/Object;)Lspace/kscience/dataforge/meta/Meta;
-}
-
-public final class space/kscience/dataforge/meta/transformations/MetaConverter$Companion {
-	public final fun getBoolean ()Lspace/kscience/dataforge/meta/transformations/MetaConverter;
-	public final fun getDouble ()Lspace/kscience/dataforge/meta/transformations/MetaConverter;
-	public final fun getFloat ()Lspace/kscience/dataforge/meta/transformations/MetaConverter;
-	public final fun getInt ()Lspace/kscience/dataforge/meta/transformations/MetaConverter;
-	public final fun getLong ()Lspace/kscience/dataforge/meta/transformations/MetaConverter;
-	public final fun getMeta ()Lspace/kscience/dataforge/meta/transformations/MetaConverter;
-	public final fun getNumber ()Lspace/kscience/dataforge/meta/transformations/MetaConverter;
-	public final fun getString ()Lspace/kscience/dataforge/meta/transformations/MetaConverter;
-	public final fun getValue ()Lspace/kscience/dataforge/meta/transformations/MetaConverter;
-	public final fun valueList (Lkotlin/jvm/functions/Function1;Lkotlin/jvm/functions/Function1;)Lspace/kscience/dataforge/meta/transformations/MetaConverter;
-	public static synthetic fun valueList$default (Lspace/kscience/dataforge/meta/transformations/MetaConverter$Companion;Lkotlin/jvm/functions/Function1;Lkotlin/jvm/functions/Function1;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/transformations/MetaConverter;
-}
-
-public final class space/kscience/dataforge/meta/transformations/MetaConverterKt {
-	public static final fun nullableMetaToObject (Lspace/kscience/dataforge/meta/transformations/MetaConverter;Lspace/kscience/dataforge/meta/Meta;)Ljava/lang/Object;
-	public static final fun nullableObjectToMeta (Lspace/kscience/dataforge/meta/transformations/MetaConverter;Ljava/lang/Object;)Lspace/kscience/dataforge/meta/Meta;
-	public static final fun valueToObject (Lspace/kscience/dataforge/meta/transformations/MetaConverter;Lspace/kscience/dataforge/meta/Value;)Ljava/lang/Object;
-}
-
-public final class space/kscience/dataforge/meta/transformations/MetaTransformation {
-	public static final field Companion Lspace/kscience/dataforge/meta/transformations/MetaTransformation$Companion;
-	public static final fun apply-impl (Ljava/util/Collection;Lspace/kscience/dataforge/meta/Meta;)Lspace/kscience/dataforge/meta/Meta;
-	public static final fun bind-impl (Ljava/util/Collection;Lspace/kscience/dataforge/meta/ObservableMeta;Lspace/kscience/dataforge/meta/MutableMeta;)V
-	public static final synthetic fun box-impl (Ljava/util/Collection;)Lspace/kscience/dataforge/meta/transformations/MetaTransformation;
-	public static fun constructor-impl (Ljava/util/Collection;)Ljava/util/Collection;
-	public fun equals (Ljava/lang/Object;)Z
-	public static fun equals-impl (Ljava/util/Collection;Ljava/lang/Object;)Z
-	public static final fun equals-impl0 (Ljava/util/Collection;Ljava/util/Collection;)Z
-	public static final fun generate-impl (Ljava/util/Collection;Lspace/kscience/dataforge/meta/Meta;)Lspace/kscience/dataforge/meta/Meta;
-	public fun hashCode ()I
-	public static fun hashCode-impl (Ljava/util/Collection;)I
-	public fun toString ()Ljava/lang/String;
-	public static fun toString-impl (Ljava/util/Collection;)Ljava/lang/String;
-	public final synthetic fun unbox-impl ()Ljava/util/Collection;
-}
-
-public final class space/kscience/dataforge/meta/transformations/MetaTransformation$Companion {
-	public final fun make-XNaMui4 (Lkotlin/jvm/functions/Function1;)Ljava/util/Collection;
-}
-
-public final class space/kscience/dataforge/meta/transformations/MetaTransformationBuilder {
-	public fun <init> ()V
-	public final fun build-050menU ()Ljava/util/Collection;
-	public final fun keep (Ljava/lang/String;)V
-	public final fun keep (Lkotlin/jvm/functions/Function1;)V
-	public final fun keep (Lspace/kscience/dataforge/names/Name;)V
-	public final fun move (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function1;)V
-	public static synthetic fun move$default (Lspace/kscience/dataforge/meta/transformations/MetaTransformationBuilder;Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function1;ILjava/lang/Object;)V
-}
-
-public final class space/kscience/dataforge/meta/transformations/RegexItemTransformationRule : space/kscience/dataforge/meta/transformations/TransformationRule {
-	public fun <init> (Lkotlin/text/Regex;Lkotlin/jvm/functions/Function4;)V
-	public final fun component1 ()Lkotlin/text/Regex;
-	public final fun component2 ()Lkotlin/jvm/functions/Function4;
-	public final fun copy (Lkotlin/text/Regex;Lkotlin/jvm/functions/Function4;)Lspace/kscience/dataforge/meta/transformations/RegexItemTransformationRule;
-	public static synthetic fun copy$default (Lspace/kscience/dataforge/meta/transformations/RegexItemTransformationRule;Lkotlin/text/Regex;Lkotlin/jvm/functions/Function4;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/transformations/RegexItemTransformationRule;
-	public fun equals (Ljava/lang/Object;)Z
-	public final fun getFrom ()Lkotlin/text/Regex;
-	public final fun getTransform ()Lkotlin/jvm/functions/Function4;
-	public fun hashCode ()I
-	public fun matches (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/Meta;)Z
-	public fun toString ()Ljava/lang/String;
-	public fun transformItem (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/Meta;Lspace/kscience/dataforge/meta/MutableMeta;)V
-}
-
-public final class space/kscience/dataforge/meta/transformations/SingleItemTransformationRule : space/kscience/dataforge/meta/transformations/TransformationRule {
-	public fun <init> (Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function3;)V
-	public final fun component1 ()Lspace/kscience/dataforge/names/Name;
-	public final fun component2 ()Lkotlin/jvm/functions/Function3;
-	public final fun copy (Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function3;)Lspace/kscience/dataforge/meta/transformations/SingleItemTransformationRule;
-	public static synthetic fun copy$default (Lspace/kscience/dataforge/meta/transformations/SingleItemTransformationRule;Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function3;ILjava/lang/Object;)Lspace/kscience/dataforge/meta/transformations/SingleItemTransformationRule;
-	public fun equals (Ljava/lang/Object;)Z
-	public final fun getFrom ()Lspace/kscience/dataforge/names/Name;
-	public final fun getTransform ()Lkotlin/jvm/functions/Function3;
-	public fun hashCode ()I
-	public fun matches (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/Meta;)Z
-	public fun selectItems (Lspace/kscience/dataforge/meta/Meta;)Lkotlin/sequences/Sequence;
-	public fun toString ()Ljava/lang/String;
-	public fun transformItem (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/Meta;Lspace/kscience/dataforge/meta/MutableMeta;)V
-}
-
-public abstract interface class space/kscience/dataforge/meta/transformations/TransformationRule {
-	public abstract fun matches (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/Meta;)Z
-	public fun selectItems (Lspace/kscience/dataforge/meta/Meta;)Lkotlin/sequences/Sequence;
-	public abstract fun transformItem (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/meta/Meta;Lspace/kscience/dataforge/meta/MutableMeta;)V
-}
-
-public final class space/kscience/dataforge/misc/CastJvmKt {
-	public static final fun unsafeCast (Ljava/lang/Object;)Ljava/lang/Object;
 }
 
 public abstract interface annotation class space/kscience/dataforge/misc/DFBuilder : java/lang/annotation/Annotation {
@@ -903,7 +972,7 @@ public abstract interface annotation class space/kscience/dataforge/misc/DFExper
 public abstract interface annotation class space/kscience/dataforge/misc/DFInternal : java/lang/annotation/Annotation {
 }
 
-public abstract interface annotation class space/kscience/dataforge/misc/DfId : java/lang/annotation/Annotation {
+public abstract interface annotation class space/kscience/dataforge/misc/DfType : java/lang/annotation/Annotation {
 	public abstract fun id ()Ljava/lang/String;
 }
 
@@ -920,6 +989,9 @@ public final class space/kscience/dataforge/misc/NamedKt {
 	public static final fun isAnonymous (Lspace/kscience/dataforge/misc/Named;)Z
 }
 
+public abstract interface annotation class space/kscience/dataforge/misc/UnsafeKType : java/lang/annotation/Annotation {
+}
+
 public final class space/kscience/dataforge/names/Name {
 	public static final field Companion Lspace/kscience/dataforge/names/Name$Companion;
 	public static final field NAME_SEPARATOR Ljava/lang/String;
@@ -937,6 +1009,16 @@ public final class space/kscience/dataforge/names/Name$Companion {
 	public final fun serializer ()Lkotlinx/serialization/KSerializer;
 }
 
+public final class space/kscience/dataforge/names/NameIndexComparator : java/util/Comparator {
+	public static final field INSTANCE Lspace/kscience/dataforge/names/NameIndexComparator;
+	public synthetic fun compare (Ljava/lang/Object;Ljava/lang/Object;)I
+	public fun compare (Ljava/lang/String;Ljava/lang/String;)I
+}
+
+public final class space/kscience/dataforge/names/NameIndexComparatorKt {
+	public static final fun getIndexedList (Lspace/kscience/dataforge/meta/Meta;Lspace/kscience/dataforge/names/Name;)Ljava/util/List;
+}
+
 public final class space/kscience/dataforge/names/NameKt {
 	public static final fun appendFirst (Lspace/kscience/dataforge/names/Name;Ljava/lang/String;)Lspace/kscience/dataforge/names/Name;
 	public static final fun appendLeft (Lspace/kscience/dataforge/names/Name;Ljava/lang/String;)Lspace/kscience/dataforge/names/Name;
@@ -944,6 +1026,7 @@ public final class space/kscience/dataforge/names/NameKt {
 	public static final fun asName (Lspace/kscience/dataforge/names/NameToken;)Lspace/kscience/dataforge/names/Name;
 	public static final fun cutFirst (Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/names/Name;
 	public static final fun cutLast (Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/names/Name;
+	public static final fun endsWith (Lspace/kscience/dataforge/names/Name;Ljava/lang/String;)Z
 	public static final fun endsWith (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/names/Name;)Z
 	public static final fun endsWith (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/names/NameToken;)Z
 	public static final fun first (Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/names/NameToken;
@@ -966,8 +1049,10 @@ public final class space/kscience/dataforge/names/NameKt {
 	public static final fun removeHeadOrNull (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/names/Name;)Lspace/kscience/dataforge/names/Name;
 	public static final fun replaceLast (Lspace/kscience/dataforge/names/Name;Lkotlin/jvm/functions/Function1;)Lspace/kscience/dataforge/names/Name;
 	public static final fun set (Ljava/util/Map;Ljava/lang/String;Ljava/lang/Object;)V
+	public static final fun startsWith (Lspace/kscience/dataforge/names/Name;Ljava/lang/String;)Z
 	public static final fun startsWith (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/names/Name;)Z
 	public static final fun startsWith (Lspace/kscience/dataforge/names/Name;Lspace/kscience/dataforge/names/NameToken;)Z
+	public static final fun toStringUnescaped (Lspace/kscience/dataforge/names/Name;)Ljava/lang/String;
 	public static final fun withIndex (Lspace/kscience/dataforge/names/Name;Ljava/lang/String;)Lspace/kscience/dataforge/names/Name;
 }
 

dataforge-meta/build.gradle.kts

@@ -6,13 +6,58 @@ kscience {
     jvm()
     js()
     native()
-    useSerialization{
+    wasm()
+    useSerialization {
         json()
     }
 }
 
 description = "Meta definition and basic operations on meta"
 
-readme{
+readme {
     maturity = space.kscience.gradle.Maturity.DEVELOPMENT
+
+    description = """
+        Core Meta and Name manipulation module
+    """.trimIndent()
+
+    feature(
+        "meta",
+        ref = "src/commonMain/kotlin/space/kscience/dataforge/meta/Meta.kt",
+        name = "Meta"
+    ) {
+        """
+        **Meta** is the representation of basic DataForge concept: Metadata, but it also could be called meta-value tree.
+        
+        Each Meta node could hava a node Value as well as a map of named child items.
+                    
+        """.trimIndent()
+    }
+
+    feature(
+        "value",
+        ref = "src/commonMain/kotlin/space/kscience/dataforge/meta/Value.kt",
+        name = "Value"
+    ) {
+        """
+        **Value** a sum type for different meta values.
+        
+        The following types are implemented in core (custom ones are also available):
+            * null
+            * boolean
+            * number
+            * string
+            * list of values
+        """.trimIndent()
+    }
+
+    feature(
+        "name",
+        ref = "src/commonMain/kotlin/space/kscience/dataforge/names/Name.kt",
+        name = "Name"
+    ) {
+        """
+        **Name** is an identifier to access tree-like structure.
+        """.trimIndent()
+    }
 }

dataforge-meta/src/commonMain/kotlin/space/kscience/dataforge/meta/JsonMeta.kt

@@ -31,12 +31,12 @@ private fun Meta.toJsonWithIndex(descriptor: MetaDescriptor?, index: String?): J
     val pairs: MutableList<Pair<String, JsonElement>> = items.entries.groupBy {
         it.key.body
     }.mapTo(ArrayList()) { (body, list) ->
-        val childDescriptor = descriptor?.children?.get(body)
+        val childDescriptor = descriptor?.nodes?.get(body)
         if (list.size == 1) {
             val (token, element) = list.first()
-                //do not add an empty element
+            //do not add an empty element
-                val child: JsonElement = element.toJsonWithIndex(childDescriptor, token.index)
+            val child: JsonElement = element.toJsonWithIndex(childDescriptor, token.index)
-            if(token.index == null) {
+            if (token.index == null) {
                 body to child
             } else {
                 body to JsonArray(listOf(child))
@@ -106,7 +106,7 @@ private fun JsonElement.toValueOrNull(descriptor: MetaDescriptor?): Value? = whe
 private fun MutableMap<NameToken, SealedMeta>.addJsonElement(
     key: String,
     element: JsonElement,
-    descriptor: MetaDescriptor?
+    descriptor: MetaDescriptor?,
 ) {
     when (element) {
         is JsonPrimitive -> put(NameToken(key), Meta(element.toValue(descriptor)))
@@ -117,8 +117,11 @@ private fun MutableMap<NameToken, SealedMeta>.addJsonElement(
             } else {
                 val indexKey = descriptor?.indexKey ?: Meta.INDEX_KEY
                 element.forEachIndexed { serial, childElement ->
-                    val index = (childElement as? JsonObject)?.get(indexKey)?.jsonPrimitive?.content
+
+                    val index = (childElement as? JsonObject)
+                        ?.get(indexKey)?.jsonPrimitive?.content
                         ?: serial.toString()
+
                     val child: SealedMeta = when (childElement) {
                         is JsonObject -> childElement.toMeta(descriptor)
                         is JsonArray -> {
@@ -133,12 +136,14 @@ private fun MutableMap<NameToken, SealedMeta>.addJsonElement(
                                 Meta(childValue)
                             }
                         }
+
                         is JsonPrimitive -> Meta(childElement.toValue(null))
                     }
                     put(NameToken(key, index), child)
                 }
             }
         }
+
         is JsonObject -> {
             val indexKey = descriptor?.indexKey ?: Meta.INDEX_KEY
             val index = element[indexKey]?.jsonPrimitive?.content
@@ -160,11 +165,15 @@ public fun JsonObject.toMeta(descriptor: MetaDescriptor? = null): SealedMeta {
 public fun JsonElement.toMeta(descriptor: MetaDescriptor? = null): SealedMeta = when (this) {
     is JsonPrimitive -> Meta(toValue(descriptor))
     is JsonObject -> toMeta(descriptor)
-    is JsonArray -> SealedMeta(null,
+    is JsonArray -> if (all { it is JsonPrimitive }) {
-        linkedMapOf<NameToken, SealedMeta>().apply {
+        Meta(map { it.toValueOrNull(descriptor) ?: error("Unreachable: should not contain objects") }.asValue())
-            addJsonElement(Meta.JSON_ARRAY_KEY, this@toMeta, null)
+    } else {
-        }
+        SealedMeta(null,
-    )
+            linkedMapOf<NameToken, SealedMeta>().apply {
+                addJsonElement(Meta.JSON_ARRAY_KEY, this@toMeta, null)
+            }
+        )
+    }
 }
 
 //

dataforge-meta/src/commonMain/kotlin/space/kscience/dataforge/meta/Laminate.kt

@@ -9,6 +9,8 @@ import space.kscience.dataforge.names.NameToken
  */
 public class Laminate internal constructor(public val layers: List<Meta>) : TypedMeta<Laminate> {
 
+    override val self: Laminate get() = this
+
     override val value: Value? = layers.firstNotNullOfOrNull { it.value }
 
     override val items: Map<NameToken, Laminate> by lazy {

dataforge-meta/src/commonMain/kotlin/space/kscience/dataforge/meta/Meta.kt

@@ -2,8 +2,7 @@ package space.kscience.dataforge.meta
 
 import kotlinx.serialization.Serializable
 import kotlinx.serialization.json.Json
-import space.kscience.dataforge.misc.DfId
+import space.kscience.dataforge.misc.DfType
-import space.kscience.dataforge.misc.unsafeCast
 import space.kscience.dataforge.names.*
 import kotlin.jvm.JvmName
 
@@ -31,7 +30,7 @@ public fun interface MetaProvider : ValueProvider {
  * TODO add documentation
  * Same name siblings are supported via elements with the same [Name] but different indices.
  */
-@DfId(Meta.TYPE)
+@DfType(Meta.TYPE)
 @Serializable(MetaSerializer::class)
 public interface Meta : MetaRepr, MetaProvider {
     public val value: Value?
@@ -151,6 +150,8 @@ public interface TypedMeta<out M : TypedMeta<M>> : Meta {
 
     override val items: Map<NameToken, M>
 
+    public val self: M
+
     override fun get(name: Name): M? {
         tailrec fun M.find(name: Name): M? = if (name.isEmpty()) {
             this
@@ -164,11 +165,6 @@ public interface TypedMeta<out M : TypedMeta<M>> : Meta {
     override fun toMeta(): Meta = this
 }
 
-/**
- * Access self as a recursive type instance
- */
-public inline val <M : TypedMeta<M>> TypedMeta<M>.self: M get() = unsafeCast()
-
 //public typealias Meta = TypedMeta<*>
 
 public operator fun <M : TypedMeta<M>> TypedMeta<M>?.get(token: NameToken): M? = this?.items?.get(token)
@@ -188,10 +184,12 @@ public operator fun <M : TypedMeta<M>> M?.get(key: String): M? = this?.get(key.p
 
 
 /**
- * Get a sequence of [Name]-[Value] pairs using top-down traversal of the tree
+ * Get a sequence of [Name]-[Value] pairs using top-down traversal of the tree.
+ * The sequence includes root value with empty name
  */
 public fun Meta.valueSequence(): Sequence<Pair<Name, Value>> = sequence {
     items.forEach { (key, item) ->
+        value?.let { yield(Name.EMPTY to it) }
         item.value?.let { itemValue ->
             yield(key.asName() to itemValue)
         }
@@ -248,7 +246,7 @@ public inline fun <reified E : Enum<E>> Meta?.enum(): E? = this?.value?.let {
     }
 }
 
-public val Meta.stringList: List<String>? get() = value?.list?.map { it.string }
+public val Meta?.stringList: List<String>? get() = this?.value?.list?.map { it.string }
 
 /**
  * Create a provider that uses given provider for default values if those are not found in this provider

dataforge-meta/src/commonMain/kotlin/space/kscience/dataforge/meta/MetaConverter.kt

@@ -0,0 +1,165 @@
+package space.kscience.dataforge.meta
+
+import kotlinx.serialization.KSerializer
+import kotlinx.serialization.json.Json
+import kotlinx.serialization.json.encodeToJsonElement
+import kotlinx.serialization.serializer
+import space.kscience.dataforge.meta.descriptors.MetaDescriptor
+import space.kscience.dataforge.misc.DFExperimental
+
+
+/**
+ * A converter of generic object to and from [Meta]
+ */
+public interface MetaConverter<T> : MetaReader<T> {
+
+    /**
+     * A descriptor for resulting meta
+     */
+    override val descriptor: MetaDescriptor? get() = null
+
+    /**
+     * Attempt conversion of [source] to an object or return null if conversion failed
+     */
+    override fun readOrNull(source: Meta): T?
+
+    override fun read(source: Meta): T =
+        readOrNull(source) ?: error("Meta $source could not be interpreted by $this")
+
+    public fun convert(obj: T): Meta
+
+    public companion object {
+
+        public val meta: MetaConverter<Meta> = object : MetaConverter<Meta> {
+            override fun readOrNull(source: Meta): Meta = source
+            override fun convert(obj: Meta): Meta = obj
+        }
+
+        public val value: MetaConverter<Value> = object : MetaConverter<Value> {
+            override fun readOrNull(source: Meta): Value? = source.value
+            override fun convert(obj: Value): Meta = Meta(obj)
+        }
+
+        public val string: MetaConverter<String> = object : MetaConverter<String> {
+            override val descriptor: MetaDescriptor = MetaDescriptor {
+                valueType(ValueType.STRING)
+            }
+
+
+            override fun readOrNull(source: Meta): String? = source.string
+            override fun convert(obj: String): Meta = Meta(obj.asValue())
+        }
+
+        public val boolean: MetaConverter<Boolean> = object : MetaConverter<Boolean> {
+            override val descriptor: MetaDescriptor = MetaDescriptor {
+                valueType(ValueType.BOOLEAN)
+            }
+
+            override fun readOrNull(source: Meta): Boolean? = source.boolean
+            override fun convert(obj: Boolean): Meta = Meta(obj.asValue())
+        }
+
+        public val number: MetaConverter<Number> = object : MetaConverter<Number> {
+            override val descriptor: MetaDescriptor = MetaDescriptor {
+                valueType(ValueType.NUMBER)
+            }
+
+            override fun readOrNull(source: Meta): Number? = source.number
+            override fun convert(obj: Number): Meta = Meta(obj.asValue())
+        }
+
+        public val double: MetaConverter<Double> = object : MetaConverter<Double> {
+            override val descriptor: MetaDescriptor = MetaDescriptor {
+                valueType(ValueType.NUMBER)
+            }
+
+            override fun readOrNull(source: Meta): Double? = source.double
+            override fun convert(obj: Double): Meta = Meta(obj.asValue())
+        }
+
+        public val float: MetaConverter<Float> = object : MetaConverter<Float> {
+            override val descriptor: MetaDescriptor = MetaDescriptor {
+                valueType(ValueType.NUMBER)
+            }
+
+            override fun readOrNull(source: Meta): Float? = source.float
+            override fun convert(obj: Float): Meta = Meta(obj.asValue())
+        }
+
+        public val int: MetaConverter<Int> = object : MetaConverter<Int> {
+            override val descriptor: MetaDescriptor = MetaDescriptor {
+                valueType(ValueType.NUMBER)
+            }
+
+            override fun readOrNull(source: Meta): Int? = source.int
+            override fun convert(obj: Int): Meta = Meta(obj.asValue())
+        }
+
+        public val long: MetaConverter<Long> = object : MetaConverter<Long> {
+            override val descriptor: MetaDescriptor = MetaDescriptor {
+                valueType(ValueType.NUMBER)
+            }
+
+            override fun readOrNull(source: Meta): Long? = source.long
+            override fun convert(obj: Long): Meta = Meta(obj.asValue())
+        }
+
+        public inline fun <reified E : Enum<E>> enum(): MetaConverter<E> = object : MetaConverter<E> {
+            override val descriptor: MetaDescriptor = MetaDescriptor {
+                valueType(ValueType.STRING)
+                allowedValues(enumValues<E>())
+            }
+
+            @Suppress("USELESS_CAST")
+            override fun readOrNull(source: Meta): E = source.enum<E>() as? E ?: error("The Item is not a Enum")
+
+            override fun convert(obj: E): Meta = Meta(obj.asValue())
+        }
+
+        public val stringList: MetaConverter<List<String>> = object : MetaConverter<List<String>> {
+            override fun convert(obj: List<String>): Meta = Meta(obj.map { it.asValue() }.asValue())
+
+            override fun readOrNull(source: Meta): List<String>? = source.stringList
+        }
+
+        public fun <T> valueList(
+            writer: (T) -> Value = { Value.of(it) },
+            reader: (Value) -> T,
+        ): MetaConverter<List<T>> = object : MetaConverter<List<T>> {
+            override val descriptor: MetaDescriptor = MetaDescriptor {
+                valueType(ValueType.LIST)
+            }
+
+            override fun readOrNull(source: Meta): List<T>? = source.value?.list?.map(reader)
+
+            override fun convert(obj: List<T>): Meta = Meta(obj.map(writer).asValue())
+        }
+
+        /**
+         * Automatically generate [MetaConverter] for a class using its serializer and optional [descriptor]
+         */
+        @DFExperimental
+        public inline fun <reified T> serializable(
+            descriptor: MetaDescriptor? = null,
+            jsonEncoder: Json = Json,
+        ): MetaConverter<T> = object : MetaConverter<T> {
+            private val serializer: KSerializer<T> = serializer()
+
+            override fun readOrNull(source: Meta): T? {
+                val json = source.toJson(descriptor)
+                return jsonEncoder.decodeFromJsonElement(serializer, json)
+            }
+
+            override fun convert(obj: T): Meta {
+                val json = jsonEncoder.encodeToJsonElement(obj)
+                return json.toMeta(descriptor)
+            }
+
+        }
+
+    }
+}
+
+public fun <T : Any> MetaConverter<T>.convertNullable(obj: T?): Meta? = obj?.let { convert(it) }
+
+

dataforge-meta/src/commonMain/kotlin/space/kscience/dataforge/meta/MetaDelegate.kt

@@ -1,84 +1,190 @@
 package space.kscience.dataforge.meta
 
-import space.kscience.dataforge.meta.transformations.MetaConverter
+import space.kscience.dataforge.meta.descriptors.Described
+import space.kscience.dataforge.meta.descriptors.MetaDescriptor
+import space.kscience.dataforge.misc.DFExperimental
 import space.kscience.dataforge.names.Name
 import space.kscience.dataforge.names.asName
 import kotlin.properties.ReadOnlyProperty
+import kotlin.reflect.KProperty
 
 /* Meta delegates */
 
-public fun MetaProvider.node(key: Name? = null): ReadOnlyProperty<Any?, Meta?> = ReadOnlyProperty { _, property ->
+public interface MetaDelegate<T> : ReadOnlyProperty<Any?, T>, Described
-    get(key ?: property.name.asName())
+
+
+public fun MetaProvider.node(
+    key: Name? = null,
+    descriptor: MetaDescriptor? = null,
+): MetaDelegate<Meta?> = object : MetaDelegate<Meta?> {
+    override val descriptor: MetaDescriptor? = descriptor
+
+    override fun getValue(thisRef: Any?, property: KProperty<*>): Meta? {
+        return get(key ?: property.name.asName())
+    }
 }
 
+
+/**
+ * Use [reader] to read the Meta node
+ */
+public fun <T> MetaProvider.readable(
+    reader: MetaReader<T>,
+    key: Name? = null,
+): MetaDelegate<T?> = object : MetaDelegate<T?> {
+    override val descriptor: MetaDescriptor? get() = reader.descriptor
+
+    override fun getValue(thisRef: Any?, property: KProperty<*>): T? {
+        return get(key ?: property.name.asName())?.let { reader.read(it) }
+    }
+}
+
+/**
+ * Use [reader] to read the Meta node or return [default] if node does not exist
+ */
+public fun <T> MetaProvider.readable(
+    reader: MetaReader<T>,
+    default: T,
+    key: Name? = null,
+): MetaDelegate<T> = object : MetaDelegate<T> {
+    override val descriptor: MetaDescriptor? get() = reader.descriptor
+
+    override fun getValue(thisRef: Any?, property: KProperty<*>): T {
+        return get(key ?: property.name.asName())?.let { reader.read(it) } ?: default
+    }
+}
+
+/**
+ * Use [reader] to read the Meta node
+ */
+@Deprecated("Replace with readable", ReplaceWith("readable(metaReader, key)"))
+public fun <T> MetaProvider.spec(
+    reader: MetaReader<T>,
+    key: Name? = null,
+): MetaDelegate<T?> = readable(reader, key)
+
+/**
+ * Use object serializer to transform it to Meta and back
+ */
+@DFExperimental
+public inline fun <reified T> MetaProvider.serializable(
+    key: Name? = null,
+    descriptor: MetaDescriptor? = null,
+): MetaDelegate<T?> = readable(MetaConverter.serializable(descriptor), key)
+
+@DFExperimental
+public inline fun <reified T> MetaProvider.serializable(
+    key: Name? = null,
+    default: T,
+    descriptor: MetaDescriptor? = null,
+): MetaDelegate<T> = readable(MetaConverter.serializable(descriptor), default, key)
+
+@Deprecated("Use convertable", ReplaceWith("convertable(converter, key)"))
 public fun <T> MetaProvider.node(
     key: Name? = null,
-    converter: MetaConverter<T>
+    converter: MetaReader<T>,
-): ReadOnlyProperty<Any?, T?> = ReadOnlyProperty { _, property ->
+): ReadOnlyProperty<Any?, T?> = readable(converter, key)
-    get(key ?: property.name.asName())?.let { converter.metaToObject(it) }
+
+/**
+ * Use [reader] to convert a list of same name siblings meta to object
+ */
+public fun <T> Meta.listOfReadable(
+    reader: MetaReader<T>,
+    key: Name? = null,
+): MetaDelegate<List<T>> = object : MetaDelegate<List<T>> {
+    override fun getValue(thisRef: Any?, property: KProperty<*>): List<T> {
+        val name = key ?: property.name.asName()
+        return getIndexed(name).values.map { reader.read(it) }
+    }
+
+    override val descriptor: MetaDescriptor? = reader.descriptor?.copy(multiple = true)
 }
 
+
+/**
+ * Use [converter] to convert a list of same name siblings meta to object
+ */
+@Deprecated("Replace with readingList", ReplaceWith("readingList(converter, key)"))
+public fun <T> Meta.listOfSpec(
+    converter: MetaReader<T>,
+    key: Name? = null,
+): MetaDelegate<List<T>> = listOfReadable(converter, key)
+
+@DFExperimental
+public inline fun <reified T> Meta.listOfSerializable(
+    key: Name? = null,
+    descriptor: MetaDescriptor? = null,
+): MetaDelegate<List<T>> = listOfReadable(MetaConverter.serializable(descriptor), key)
+
 /**
  * A property delegate that uses custom key
  */
-public fun MetaProvider.value(key: Name? = null): ReadOnlyProperty<Any?, Value?> = ReadOnlyProperty { _, property ->
+public fun MetaProvider.value(
-    get(key ?: property.name.asName())?.value
+    key: Name? = null,
+    descriptor: MetaDescriptor? = null,
+): MetaDelegate<Value?> = object : MetaDelegate<Value?> {
+    override fun getValue(thisRef: Any?, property: KProperty<*>): Value? = get(key ?: property.name.asName())?.value
+
+    override val descriptor: MetaDescriptor? = descriptor
 }
 
 public fun <R> MetaProvider.value(
     key: Name? = null,
-    reader: (Value?) -> R
+    descriptor: MetaDescriptor? = null,
-): ReadOnlyProperty<Any?, R> = ReadOnlyProperty { _, property ->
+    reader: (Value?) -> R,
-    reader(get(key ?: property.name.asName())?.value)
+): MetaDelegate<R> = object : MetaDelegate<R> {
+    override fun getValue(thisRef: Any?, property: KProperty<*>): R = reader(get(key ?: property.name.asName())?.value)
+
+    override val descriptor: MetaDescriptor? = descriptor
 }
 
 //TODO add caching for sealed nodes
 
 /* Read-only delegates for [Meta] */
 
-public fun MetaProvider.string(key: Name? = null): ReadOnlyProperty<Any?, String?> = value(key) { it?.string }
+public fun MetaProvider.string(key: Name? = null): MetaDelegate<String?> = value(key = key) { it?.string }
 
-public fun MetaProvider.boolean(key: Name? = null): ReadOnlyProperty<Any?, Boolean?> = value(key) { it?.boolean }
+public fun MetaProvider.boolean(key: Name? = null): MetaDelegate<Boolean?> = value(key = key) { it?.boolean }
 
-public fun MetaProvider.number(key: Name? = null): ReadOnlyProperty<Any?, Number?> = value(key) { it?.numberOrNull }
+public fun MetaProvider.number(key: Name? = null): MetaDelegate<Number?> = value(key = key) { it?.numberOrNull }
 
-public fun MetaProvider.double(key: Name? = null): ReadOnlyProperty<Any?, Double?> = value(key) { it?.double }
+public fun MetaProvider.double(key: Name? = null): MetaDelegate<Double?> = value(key = key) { it?.double }
 
-public fun MetaProvider.float(key: Name? = null): ReadOnlyProperty<Any?, Float?> = value(key) { it?.float }
+public fun MetaProvider.float(key: Name? = null): MetaDelegate<Float?> = value(key = key) { it?.float }
 
-public fun MetaProvider.int(key: Name? = null): ReadOnlyProperty<Any?, Int?> = value(key) { it?.int }
+public fun MetaProvider.int(key: Name? = null): MetaDelegate<Int?> = value(key = key) { it?.int }
 
-public fun MetaProvider.long(key: Name? = null): ReadOnlyProperty<Any?, Long?> = value(key) { it?.long }
+public fun MetaProvider.long(key: Name? = null): MetaDelegate<Long?> = value(key = key) { it?.long }
 
-public fun MetaProvider.string(default: String, key: Name? = null): ReadOnlyProperty<Any?, String> =
+public fun MetaProvider.string(default: String, key: Name? = null): MetaDelegate<String> =
-    value(key) { it?.string ?: default }
+    value(key = key) { it?.string ?: default }
 
-public fun MetaProvider.boolean(default: Boolean, key: Name? = null): ReadOnlyProperty<Any?, Boolean> =
+public fun MetaProvider.boolean(default: Boolean, key: Name? = null): MetaDelegate<Boolean> =
-    value(key) { it?.boolean ?: default }
+    value(key = key) { it?.boolean ?: default }
 
-public fun MetaProvider.number(default: Number, key: Name? = null): ReadOnlyProperty<Any?, Number> =
+public fun MetaProvider.number(default: Number, key: Name? = null): MetaDelegate<Number> =
-    value(key) { it?.numberOrNull ?: default }
+    value(key = key) { it?.numberOrNull ?: default }
 
-public fun MetaProvider.double(default: Double, key: Name? = null): ReadOnlyProperty<Any?, Double> =
+public fun MetaProvider.double(default: Double, key: Name? = null): MetaDelegate<Double> =
-    value(key) { it?.double ?: default }
+    value(key = key) { it?.double ?: default }
 
-public fun MetaProvider.float(default: Float, key: Name? = null): ReadOnlyProperty<Any?, Float> =
+public fun MetaProvider.float(default: Float, key: Name? = null): MetaDelegate<Float> =
-    value(key) { it?.float ?: default }
+    value(key = key) { it?.float ?: default }
 
-public fun MetaProvider.int(default: Int, key: Name? = null): ReadOnlyProperty<Any?, Int> =
+public fun MetaProvider.int(default: Int, key: Name? = null): MetaDelegate<Int> =
-    value(key) { it?.int ?: default }
+    value(key = key) { it?.int ?: default }
 
-public fun MetaProvider.long(default: Long, key: Name? = null): ReadOnlyProperty<Any?, Long> =
+public fun MetaProvider.long(default: Long, key: Name? = null): MetaDelegate<Long> =
-    value(key) { it?.long ?: default }
+    value(key = key) { it?.long ?: default }
 
-public inline fun <reified E : Enum<E>> MetaProvider.enum(default: E, key: Name? = null): ReadOnlyProperty<Any?, E> =
+public inline fun <reified E : Enum<E>> MetaProvider.enum(default: E, key: Name? = null): MetaDelegate<E> =
-    value<E>(key) { it?.enum<E>() ?: default }
+    value<E>(key = key) { it?.enum<E>() ?: default }
 
-public fun MetaProvider.string(key: Name? = null, default: () -> String): ReadOnlyProperty<Any?, String> =
+public fun MetaProvider.string(key: Name? = null, default: () -> String): MetaDelegate<String> =
-    value(key) { it?.string ?: default() }
+    value(key = key) { it?.string ?: default() }
 
-public fun MetaProvider.boolean(key: Name? = null, default: () -> Boolean): ReadOnlyProperty<Any?, Boolean> =
+public fun MetaProvider.boolean(key: Name? = null, default: () -> Boolean): MetaDelegate<Boolean> =
-    value(key) { it?.boolean ?: default() }
+    value(key = key) { it?.boolean ?: default() }
 
-public fun MetaProvider.number(key: Name? = null, default: () -> Number): ReadOnlyProperty<Any?, Number> =
+public fun MetaProvider.number(key: Name? = null, default: () -> Number): MetaDelegate<Number> =
-    value(key) { it?.numberOrNull ?: default() }
+    value(key = key) { it?.numberOrNull ?: default() }

dataforge-meta/src/commonMain/kotlin/space/kscience/dataforge/meta/MetaReader.kt

@@ -0,0 +1,21 @@
+package space.kscience.dataforge.meta
+
+import space.kscience.dataforge.meta.descriptors.Described
+
+public interface MetaReader<out T> : Described {
+
+    /**
+     * Read the source meta into an object and return null if Meta could not be interpreted as a target type
+     */
+    public fun readOrNull(source: Meta): T?
+
+    /**
+     * Read generic read-only meta with this [MetaReader] producing instance of the desired type.
+     * Throws an error if conversion could not be done.
+     */
+    public fun read(source: Meta): T = readOrNull(source) ?: error("Meta $source could not be interpreted by $this")
+}
+
+
+public fun <T : Any> MetaReader<T>.readNullable(item: Meta?): T? = item?.let { read(it) }
+public fun <T> MetaReader<T>.readValue(value: Value): T? = read(Meta(value))

dataforge-meta/src/commonMain/kotlin/space/kscience/dataforge/meta/MetaRef.kt

@@ -0,0 +1,221 @@
+package space.kscience.dataforge.meta
+
+import kotlinx.serialization.json.Json
+import space.kscience.dataforge.meta.descriptors.Described
+import space.kscience.dataforge.meta.descriptors.MetaDescriptor
+import space.kscience.dataforge.meta.descriptors.MetaDescriptorBuilder
+import space.kscience.dataforge.misc.DFExperimental
+import space.kscience.dataforge.names.Name
+import space.kscience.dataforge.names.asName
+import space.kscience.dataforge.names.startsWith
+import kotlin.properties.PropertyDelegateProvider
+import kotlin.properties.ReadOnlyProperty
+
+
+/**
+ * A reference to a read-only value of type [T] inside [MetaProvider] or writable value in [MutableMetaProvider]
+ */
+@DFExperimental
+public data class MetaRef<T>(
+    public val name: Name,
+    public val converter: MetaConverter<T>,
+    override val descriptor: MetaDescriptor? = converter.descriptor,
+) : Described
+
+/**
+ * Get a value from provider by [ref] or return null if node with given name is missing
+ */
+@DFExperimental
+public operator fun <T> MetaProvider.get(ref: MetaRef<T>): T? = get(ref.name)?.let { ref.converter.readOrNull(it) }
+
+/**
+ * Set a value in a mutable provider by [ref]
+ */
+@DFExperimental
+public operator fun <T> MutableMetaProvider.set(ref: MetaRef<T>, value: T) {
+    set(ref.name, ref.converter.convert(value))
+}
+
+/**
+ * Observe changes to specific property via given [ref].
+ *
+ * This listener should be removed in a same way as [ObservableMeta.onChange].
+ *
+ * @param callback an action to be performed on each change of item. Null means that the item is not present or malformed.
+ */
+@DFExperimental
+public fun <T: Any> ObservableMeta.onValueChange(owner: Any?, ref: MetaRef<T>, callback: (T?) -> Unit) {
+    onChange(owner) { name ->
+        if (name.startsWith(ref.name)) {
+            get(name)?.let { value ->
+                callback(ref.converter.readOrNull(value))
+            }
+        }
+    }
+}
+
+/**
+ * Remove a node corresponding to [ref] from a mutable provider if it exists
+ */
+@DFExperimental
+public fun MutableMetaProvider.remove(ref: MetaRef<*>) {
+    remove(ref.name)
+}
+
+/**
+ * Base storage of [MetaRef]
+ */
+@OptIn(DFExperimental::class)
+public interface MetaRefStore : Described {
+    public val refs: List<MetaRef<*>>
+}
+
+/**
+ * A base class for [Meta] specification that stores references to meta nodes.
+ */
+@DFExperimental
+public abstract class MetaSpec : MetaRefStore {
+    private val _refs: MutableList<MetaRef<*>> = mutableListOf()
+    override val refs: List<MetaRef<*>> get() = _refs
+
+    /**
+     * Register a ref in this specification
+     */
+    protected fun registerRef(ref: MetaRef<*>) {
+        _refs.add(ref)
+    }
+
+    /**
+     * Create and register a ref by property name and provided converter.
+     * By default, uses descriptor from the converter
+     */
+    public fun <T> item(
+        converter: MetaConverter<T>,
+        key: Name? = null,
+        descriptor: MetaDescriptor? = converter.descriptor,
+    ): PropertyDelegateProvider<MetaSpec, ReadOnlyProperty<MetaSpec, MetaRef<T>>> =
+        PropertyDelegateProvider { _, property ->
+            val ref = MetaRef(key ?: property.name.asName(), converter, descriptor)
+            registerRef(ref)
+            ReadOnlyProperty { _, _ ->
+                ref
+            }
+        }
+
+    /**
+     * Override to provide custom [MetaDescriptor]
+     */
+    protected open fun MetaDescriptorBuilder.buildDescriptor(): Unit = Unit
+
+    override val descriptor: MetaDescriptor by lazy {
+        MetaDescriptor {
+            refs.forEach { ref ->
+                ref.descriptor?.let {
+                    node(ref.name, ref.descriptor)
+                }
+            }
+            buildDescriptor()
+        }
+    }
+}
+
+/**
+ * Register an item using a [descriptorBuilder] to customize descriptor
+ */
+@DFExperimental
+public fun <T> MetaSpec.item(
+    converter: MetaConverter<T>,
+    key: Name? = null,
+    descriptorBuilder: MetaDescriptorBuilder.() -> Unit = {},
+): PropertyDelegateProvider<MetaSpec, ReadOnlyProperty<MetaSpec, MetaRef<T>>> = item(converter, key, MetaDescriptor {
+    converter.descriptor?.let { from(it) }
+    descriptorBuilder()
+})
+
+//utility methods to add different nodes
+
+@DFExperimental
+public fun MetaSpec.metaItem(
+    key: Name? = null,
+    descriptorBuilder: MetaDescriptorBuilder.() -> Unit = {},
+): PropertyDelegateProvider<MetaSpec, ReadOnlyProperty<MetaSpec, MetaRef<Meta>>> =
+    item(MetaConverter.meta, key, descriptorBuilder)
+
+@DFExperimental
+public fun MetaSpec.string(
+    key: Name? = null,
+    descriptorBuilder: MetaDescriptorBuilder.() -> Unit = {},
+): PropertyDelegateProvider<MetaSpec, ReadOnlyProperty<MetaSpec, MetaRef<String>>> =
+    item(MetaConverter.string, key, descriptorBuilder)
+
+@DFExperimental
+public fun MetaSpec.boolean(
+    key: Name? = null,
+    descriptorBuilder: MetaDescriptorBuilder.() -> Unit = {},
+): PropertyDelegateProvider<MetaSpec, ReadOnlyProperty<MetaSpec, MetaRef<Boolean>>> =
+    item(MetaConverter.boolean, key, descriptorBuilder)
+
+@DFExperimental
+public fun MetaSpec.stringList(
+    key: Name? = null,
+    descriptorBuilder: MetaDescriptorBuilder.() -> Unit = {},
+): PropertyDelegateProvider<MetaSpec, ReadOnlyProperty<MetaSpec, MetaRef<List<String>>>> =
+    item(MetaConverter.stringList, key, descriptorBuilder)
+
+@DFExperimental
+public fun MetaSpec.float(
+    key: Name? = null,
+    descriptorBuilder: MetaDescriptorBuilder.() -> Unit = {},
+): PropertyDelegateProvider<MetaSpec, ReadOnlyProperty<MetaSpec, MetaRef<Float>>> =
+    item(MetaConverter.float, key, descriptorBuilder)
+
+@DFExperimental
+public fun MetaSpec.double(
+    key: Name? = null,
+    descriptorBuilder: MetaDescriptorBuilder.() -> Unit = {},
+): PropertyDelegateProvider<MetaSpec, ReadOnlyProperty<MetaSpec, MetaRef<Double>>> =
+    item(MetaConverter.double, key, descriptorBuilder)
+
+@DFExperimental
+public fun MetaSpec.int(
+    key: Name? = null,
+    descriptorBuilder: MetaDescriptorBuilder.() -> Unit = {},
+): PropertyDelegateProvider<MetaSpec, ReadOnlyProperty<MetaSpec, MetaRef<Int>>> =
+    item(MetaConverter.int, key, descriptorBuilder)
+
+@DFExperimental
+public fun MetaSpec.long(
+    key: Name? = null,
+    descriptorBuilder: MetaDescriptorBuilder.() -> Unit = {},
+): PropertyDelegateProvider<MetaSpec, ReadOnlyProperty<MetaSpec, MetaRef<Long>>> =
+    item(MetaConverter.long, key, descriptorBuilder)
+
+
+@DFExperimental
+public fun MetaSpec.doubleArray(
+    key: Name? = null,
+    descriptorBuilder: MetaDescriptorBuilder.() -> Unit = {},
+): PropertyDelegateProvider<MetaSpec, ReadOnlyProperty<MetaSpec, MetaRef<DoubleArray>>> =
+    item(MetaConverter.doubleArray, key, descriptorBuilder)
+
+@DFExperimental
+public fun MetaSpec.byteArray(
+    key: Name? = null,
+    descriptorBuilder: MetaDescriptorBuilder.() -> Unit = {},
+): PropertyDelegateProvider<MetaSpec, ReadOnlyProperty<MetaSpec, MetaRef<ByteArray>>> =
+    item(MetaConverter.byteArray, key, descriptorBuilder)
+
+@DFExperimental
+public inline fun <reified E : Enum<E>> MetaSpec.enum(
+    key: Name? = null,
+    noinline descriptorBuilder: MetaDescriptorBuilder.() -> Unit = {},
+): PropertyDelegateProvider<MetaSpec, ReadOnlyProperty<MetaSpec, MetaRef<E>>> =
+    item(MetaConverter.enum(), key, descriptorBuilder)
+
+@DFExperimental
+public inline fun <reified T> MetaSpec.serializable(
+    key: Name? = null,
+    jsonEncoder: Json = Json,
+    noinline descriptorBuilder: MetaDescriptorBuilder.() -> Unit = {},
+): PropertyDelegateProvider<MetaSpec, ReadOnlyProperty<MetaSpec, MetaRef<T>>> =
+    item(MetaConverter.serializable(jsonEncoder = jsonEncoder), key, descriptorBuilder)

dataforge-meta/src/commonMain/kotlin/space/kscience/dataforge/meta/MetaSerializer.kt

@@ -45,4 +45,21 @@ public object MutableMetaSerializer : KSerializer<MutableMeta> {
     override fun serialize(encoder: Encoder, value: MutableMeta) {
         encoder.encodeSerializableValue(MetaSerializer, value)
     }
+}
+
+/**
+ * A serializer for [ObservableMutableMeta]
+ */
+public object ObservableMutableMetaSerializer : KSerializer<ObservableMutableMeta> {
+
+    override val descriptor: SerialDescriptor = MetaSerializer.descriptor
+
+    override fun deserialize(decoder: Decoder): ObservableMutableMeta {
+        val meta = decoder.decodeSerializableValue(MetaSerializer)
+        return ((meta as? MutableMeta) ?: meta.toMutableMeta()).asObservable()
+    }
+
+    override fun serialize(encoder: Encoder, value: ObservableMutableMeta) {
+        encoder.encodeSerializableValue(MetaSerializer, value)
+    }
 }

dataforge-meta/src/commonMain/kotlin/space/kscience/dataforge/meta/MetaTransformation.kt

@@ -1,6 +1,5 @@
-package space.kscience.dataforge.meta.transformations
+package space.kscience.dataforge.meta
 
-import space.kscience.dataforge.meta.*
 import space.kscience.dataforge.misc.DFExperimental
 import space.kscience.dataforge.names.Name
 import kotlin.jvm.JvmInline

dataforge-meta/src/commonMain/kotlin/space/kscience/dataforge/meta/MutableMeta.kt

@@ -19,6 +19,10 @@ public annotation class MetaBuilderMarker
 public interface MutableMetaProvider : MetaProvider, MutableValueProvider {
     override fun get(name: Name): MutableMeta?
     public operator fun set(name: Name, node: Meta?)
+
+    /**
+     * Set value with the given name. Does nothing if value is not changed.
+     */
     override fun setValue(name: Name, value: Value?)
 }
 
@@ -48,11 +52,13 @@ public interface MutableMeta : Meta, MutableMetaProvider {
     }
 
     override fun setValue(name: Name, value: Value?) {
-        getOrCreate(name).value = value
+        if (value != getValue(name)) {
+            getOrCreate(name).value = value
+        }
     }
 
     /**
-     * Get existing node or create a new one
+     * Get an existing node or create a new one
      */
     public fun getOrCreate(name: Name): MutableMeta
 
@@ -122,6 +128,10 @@ public interface MutableMeta : Meta, MutableMetaProvider {
         setValue(Name.parse(this), array.asValue())
     }
 
+    public infix fun String.put(array: ByteArray) {
+        setValue(Name.parse(this), array.asValue())
+    }
+
     public infix fun String.put(repr: MetaRepr) {
         set(Name.parse(this), repr.toMeta())
     }
@@ -149,7 +159,17 @@ public interface MutableTypedMeta<M : MutableTypedMeta<M>> : TypedMeta<M>, Mutab
      */
     @DFExperimental
     public fun attach(name: Name, node: M)
-    override fun get(name: Name): M?
+
+    override fun get(name: Name): M? {
+        tailrec fun M.find(name: Name): M? = if (name.isEmpty()) {
+            self
+        } else {
+            items[name.firstOrNull()!!]?.find(name.cutFirst())
+        }
+
+        return self.find(name)
+    }
+
     override fun getOrCreate(name: Name): M
 }
 
@@ -165,7 +185,7 @@ public fun MutableMetaProvider.remove(key: String) {
 
 // node setters
 
-public operator fun MutableMetaProvider.set(Key: NameToken, value: Meta): Unit = set(Key.asName(), value)
+public operator fun MutableMetaProvider.set(key: NameToken, value: Meta): Unit = set(key.asName(), value)
 public operator fun MutableMetaProvider.set(key: String, value: Meta): Unit = set(Name.parse(key), value)
 
 
@@ -198,10 +218,8 @@ public operator fun MutableMetaProvider.set(key: String, metas: Iterable<Meta>):
 
 
 /**
- * Update existing mutable node with another node. The rules are following:
+ * Update the existing mutable node with another node.
- *  * value replaces anything
+ * Values that are present in the current provider and are missing in [meta] are kept.
- *  * node updates node and replaces anything but node
- *  * node list updates node list if number of nodes in the list is the same and replaces anything otherwise
  */
 public fun MutableMetaProvider.update(meta: Meta) {
     meta.valueSequence().forEach { (name, value) ->
@@ -222,7 +240,7 @@ public fun <M : MutableTypedMeta<M>> MutableTypedMeta<M>.edit(name: Name, builde
     getOrCreate(name).apply(builder)
 
 /**
- * Set a value at a given [name]. If node does not exist, create it.
+ * Set a value at a given [name]. If a node does not exist, create it.
  */
 public operator fun <M : MutableTypedMeta<M>> MutableTypedMeta<M>.set(name: Name, value: Value?) {
     edit(name) {
@@ -245,6 +263,9 @@ private class MutableMetaImpl(
     value: Value?,
     children: Map<NameToken, Meta> = emptyMap(),
 ) : AbstractObservableMeta(), ObservableMutableMeta {
+
+    override val self get() = this
+
     override var value = value
         @ThreadSafe set(value) {
             val oldValue = field
@@ -324,8 +345,6 @@ private class MutableMetaImpl(
                     //remove child and invalidate if argument is null
                     if (node == null) {
                         children.remove(token)?.removeListener(this)
-                        // old item is not null otherwise we can't be here
-                        invalidate(name)
                     } else {
                         val newNode = wrapItem(node)
                         newNode.adoptBy(this, token)
@@ -335,7 +354,7 @@ private class MutableMetaImpl(
 
                 else -> {
                     val token = name.firstOrNull()!!
-                    //get existing or create new node.
+                    //get an existing node or create a new node.
                     if (items[token] == null) {
                         val newNode = MutableMetaImpl(null)
                         newNode.adoptBy(this, token)
@@ -369,16 +388,37 @@ public fun MutableMeta.append(name: Name, value: Value): Unit = append(name, Met
 
 public fun MutableMeta.append(key: String, value: Value): Unit = append(Name.parse(key), value)
 
+/**
+ * Update all items that exist in the [newMeta] and remove existing items that are missing in [newMeta].
+ * This produces the same result as clearing all items and updating blank meta with a [newMeta], but does not
+ * produce unnecessary invalidation events (if they are supported).
+ */
+public fun MutableMeta.reset(newMeta: Meta) {
+    //remove old items
+    (items.keys - newMeta.items.keys).forEach {
+        remove(it.asName())
+    }
+    newMeta.items.forEach { (token, item) ->
+        set(token, item)
+    }
+}
+
 /**
  * Create a mutable copy of this meta. The copy is created even if the Meta is already mutable
  */
-public fun Meta.toMutableMeta(): ObservableMutableMeta = MutableMetaImpl(value, items)
+public fun Meta.toMutableMeta(): MutableMeta =
+    MutableMeta { update(this@toMutableMeta) } //MutableMetaImpl(value, items)
 
 public fun Meta.asMutableMeta(): MutableMeta = (this as? MutableMeta) ?: toMutableMeta()
 
 @JsName("newObservableMutableMeta")
 public fun ObservableMutableMeta(): ObservableMutableMeta = MutableMetaImpl(null)
 
+/**
+ * Create a pre-filled [ObservableMutableMeta]
+ */
+public fun ObservableMutableMeta(content: Meta): ObservableMutableMeta = ObservableMutableMeta { update(content) }
+
 /**
  * Build a [MutableMeta] using given transformation
  */
@@ -387,12 +427,14 @@ public inline fun ObservableMutableMeta(builder: MutableMeta.() -> Unit = {}): O
 
 
 /**
- * Create a copy of this [Meta], optionally applying the given [block].
+ * Create a read-only copy of this [Meta]. [modification] is an optional modification applied to [Meta] on copy.
- * The listeners of the original Config are not retained.
+ *
+ *  The copy does not reflect changes of the initial Meta.
  */
-public inline fun Meta.copy(block: MutableMeta.() -> Unit = {}): Meta =
+public inline fun Meta.copy(modification: MutableMeta.() -> Unit = {}): Meta = Meta {
-    toMutableMeta().apply(block)
+    update(this@copy)
-
+    modification()
+}
 
 private class MutableMetaWithDefault(
     val source: MutableMeta, val default: MetaProvider, val rootName: Name,

dataforge-meta/src/commonMain/kotlin/space/kscience/dataforge/meta/MutableMetaDelegate.kt

@@ -1,52 +1,146 @@
 package space.kscience.dataforge.meta
 
-import space.kscience.dataforge.meta.transformations.MetaConverter
+import space.kscience.dataforge.meta.descriptors.Described
+import space.kscience.dataforge.meta.descriptors.MetaDescriptor
+import space.kscience.dataforge.misc.DFExperimental
 import space.kscience.dataforge.names.Name
 import space.kscience.dataforge.names.asName
+import space.kscience.dataforge.names.getIndexedList
 import kotlin.properties.ReadWriteProperty
 import kotlin.reflect.KProperty
 
+
 /* Read-write delegates */
 
-public fun MutableMetaProvider.node(key: Name? = null): ReadWriteProperty<Any?, Meta?> =
+public interface MutableMetaDelegate<T> : ReadWriteProperty<Any?, T>, Described
-    object : ReadWriteProperty<Any?, Meta?> {
-        override fun getValue(thisRef: Any?, property: KProperty<*>): Meta? {
-            return get(key ?: property.name.asName())
-        }
 
-        override fun setValue(thisRef: Any?, property: KProperty<*>, value: Meta?) {
+public fun MutableMetaProvider.node(
-            val name = key ?: property.name.asName()
+    key: Name? = null,
-            set(name, value)
+    descriptor: MetaDescriptor? = null,
-        }
+): MutableMetaDelegate<Meta?> = object : MutableMetaDelegate<Meta?> {
+
+    override val descriptor: MetaDescriptor? = descriptor
+
+    override fun getValue(thisRef: Any?, property: KProperty<*>): Meta? {
+        return get(key ?: property.name.asName())
     }
 
-public fun <T> MutableMetaProvider.node(key: Name? = null, converter: MetaConverter<T>): ReadWriteProperty<Any?, T?> =
+    override fun setValue(thisRef: Any?, property: KProperty<*>, value: Meta?) {
-    object : ReadWriteProperty<Any?, T?> {
+        val name = key ?: property.name.asName()
-        override fun getValue(thisRef: Any?, property: KProperty<*>): T? {
+        set(name, value)
-            return get(key ?: property.name.asName())?.let { converter.metaToObject(it) }
+    }
-        }
+}
 
-        override fun setValue(thisRef: Any?, property: KProperty<*>, value: T?) {
+/**
-            val name = key ?: property.name.asName()
+ * Use [converter] to transform an object to Meta and back.
-            set(name, value?.let { converter.objectToMeta(it) })
+ * Note that mutation of the object does not change Meta.
-        }
+ */
+public fun <T> MutableMetaProvider.convertable(
+    converter: MetaConverter<T>,
+    key: Name? = null,
+): MutableMetaDelegate<T?> = object : MutableMetaDelegate<T?> {
+
+    override val descriptor: MetaDescriptor? get() = converter.descriptor
+
+
+    override fun getValue(thisRef: Any?, property: KProperty<*>): T? {
+        val name = key ?: property.name.asName()
+        return get(name)?.let { converter.read(it) }
     }
 
-public fun MutableMetaProvider.value(key: Name? = null): ReadWriteProperty<Any?, Value?> =
+    override fun setValue(thisRef: Any?, property: KProperty<*>, value: T?) {
-    object : ReadWriteProperty<Any?, Value?> {
+        val name = key ?: property.name.asName()
-        override fun getValue(thisRef: Any?, property: KProperty<*>): Value? =
+        set(name, value?.let { converter.convert(it) })
-            get(key ?: property.name.asName())?.value
-
-        override fun setValue(thisRef: Any?, property: KProperty<*>, value: Value?) {
-            setValue(key ?: property.name.asName(), value)
-        }
     }
+}
+
+public fun <T> MutableMetaProvider.convertable(
+    converter: MetaConverter<T>,
+    default: T,
+    key: Name? = null,
+): MutableMetaDelegate<T> = object : MutableMetaDelegate<T> {
+
+    override val descriptor: MetaDescriptor? get() = converter.descriptor
+
+
+    override fun getValue(thisRef: Any?, property: KProperty<*>): T {
+        val name = key ?: property.name.asName()
+        return get(name)?.let { converter.read(it) } ?: default
+    }
+
+    override fun setValue(thisRef: Any?, property: KProperty<*>, value: T) {
+        val name = key ?: property.name.asName()
+        set(name, value?.let { converter.convert(it) })
+    }
+}
+
+/**
+ * Use object serializer to transform it to Meta and back.
+ * Note that mutation of the object does not change Meta.
+ */
+@DFExperimental
+public inline fun <reified T> MutableMetaProvider.serializable(
+    descriptor: MetaDescriptor? = null,
+    key: Name? = null,
+): MutableMetaDelegate<T?> = convertable<T>(MetaConverter.serializable(descriptor), key)
+
+@DFExperimental
+public inline fun <reified T> MutableMetaProvider.serializable(
+    descriptor: MetaDescriptor? = null,
+    default: T,
+    key: Name? = null,
+): MutableMetaDelegate<T> = convertable(MetaConverter.serializable(descriptor), default, key)
+
+/**
+ * Use [converter] to convert a list of same name siblings meta to object and back.
+ * Note that mutation of the object does not change Meta.
+ */
+public fun <T> MutableMeta.listOfConvertable(
+    converter: MetaConverter<T>,
+    key: Name? = null,
+): MutableMetaDelegate<List<T>> = object : MutableMetaDelegate<List<T>> {
+    override val descriptor: MetaDescriptor? = converter.descriptor?.copy(multiple = true)
+
+    override fun getValue(thisRef: Any?, property: KProperty<*>): List<T> {
+        val name = key ?: property.name.asName()
+        return getIndexedList(name).map { converter.read(it) }
+    }
+
+    override fun setValue(thisRef: Any?, property: KProperty<*>, value: List<T>) {
+        val name = key ?: property.name.asName()
+        setIndexed(name, value.map { converter.convert(it) })
+    }
+}
+
+@DFExperimental
+public inline fun <reified T> MutableMeta.listOfSerializable(
+    key: Name? = null,
+    descriptor: MetaDescriptor? = null,
+): MutableMetaDelegate<List<T>> = listOfConvertable(MetaConverter.serializable(descriptor), key)
+
+
+public fun MutableMetaProvider.value(
+    key: Name? = null,
+    descriptor: MetaDescriptor? = null,
+): MutableMetaDelegate<Value?> = object : MutableMetaDelegate<Value?> {
+    override val descriptor: MetaDescriptor? = descriptor
+
+    override fun getValue(thisRef: Any?, property: KProperty<*>): Value? =
+        get(key ?: property.name.asName())?.value
+
+    override fun setValue(thisRef: Any?, property: KProperty<*>, value: Value?) {
+        setValue(key ?: property.name.asName(), value)
+    }
+}
 
 public fun <T> MutableMetaProvider.value(
     key: Name? = null,
     writer: (T) -> Value? = { Value.of(it) },
-    reader: (Value?) -> T
+    descriptor: MetaDescriptor? = null,
-): ReadWriteProperty<Any?, T> = object : ReadWriteProperty<Any?, T> {
+    reader: (Value?) -> T,
+): MutableMetaDelegate<T> = object : MutableMetaDelegate<T> {
+    override val descriptor: MetaDescriptor? = descriptor
+
     override fun getValue(thisRef: Any?, property: KProperty<*>): T =
         reader(get(key ?: property.name.asName())?.value)
 
@@ -57,65 +151,65 @@ public fun <T> MutableMetaProvider.value(
 
 /* Read-write delegates for [MutableItemProvider] */
 
-public fun MutableMetaProvider.string(key: Name? = null): ReadWriteProperty<Any?, String?> =
+public fun MutableMetaProvider.string(key: Name? = null): MutableMetaDelegate<String?> =
     value(key) { it?.string }
 
-public fun MutableMetaProvider.boolean(key: Name? = null): ReadWriteProperty<Any?, Boolean?> =
+public fun MutableMetaProvider.boolean(key: Name? = null): MutableMetaDelegate<Boolean?> =
     value(key) { it?.boolean }
 
-public fun MutableMetaProvider.number(key: Name? = null): ReadWriteProperty<Any?, Number?> =
+public fun MutableMetaProvider.number(key: Name? = null): MutableMetaDelegate<Number?> =
     value(key) { it?.number }
 
-public fun MutableMetaProvider.string(default: String, key: Name? = null): ReadWriteProperty<Any?, String> =
+public fun MutableMetaProvider.string(default: String, key: Name? = null): MutableMetaDelegate<String> =
     value(key) { it?.string ?: default }
 
-public fun MutableMetaProvider.boolean(default: Boolean, key: Name? = null): ReadWriteProperty<Any?, Boolean> =
+public fun MutableMetaProvider.boolean(default: Boolean, key: Name? = null): MutableMetaDelegate<Boolean> =
     value(key) { it?.boolean ?: default }
 
-public fun MutableMetaProvider.number(default: Number, key: Name? = null): ReadWriteProperty<Any?, Number> =
+public fun MutableMetaProvider.number(default: Number, key: Name? = null): MutableMetaDelegate<Number> =
     value(key) { it?.number ?: default }
 
-public fun MutableMetaProvider.string(key: Name? = null, default: () -> String): ReadWriteProperty<Any?, String> =
+public fun MutableMetaProvider.string(key: Name? = null, default: () -> String): MutableMetaDelegate<String> =
     value(key) { it?.string ?: default() }
 
-public fun MutableMetaProvider.boolean(key: Name? = null, default: () -> Boolean): ReadWriteProperty<Any?, Boolean> =
+public fun MutableMetaProvider.boolean(key: Name? = null, default: () -> Boolean): MutableMetaDelegate<Boolean> =
     value(key) { it?.boolean ?: default() }
 
-public fun MutableMetaProvider.number(key: Name? = null, default: () -> Number): ReadWriteProperty<Any?, Number> =
+public fun MutableMetaProvider.number(key: Name? = null, default: () -> Number): MutableMetaDelegate<Number> =
     value(key) { it?.number ?: default() }
 
 public inline fun <reified E : Enum<E>> MutableMetaProvider.enum(
     default: E,
     key: Name? = null,
-): ReadWriteProperty<Any?, E> = value(key) { value -> value?.string?.let { enumValueOf<E>(it) } ?: default }
+): MutableMetaDelegate<E> = value(key) { value -> value?.string?.let { enumValueOf<E>(it) } ?: default }
 
 /* Number delegates */
 
-public fun MutableMetaProvider.int(key: Name? = null): ReadWriteProperty<Any?, Int?> =
+public fun MutableMetaProvider.int(key: Name? = null): MutableMetaDelegate<Int?> =
     value(key) { it?.int }
 
-public fun MutableMetaProvider.double(key: Name? = null): ReadWriteProperty<Any?, Double?> =
+public fun MutableMetaProvider.double(key: Name? = null): MutableMetaDelegate<Double?> =
     value(key) { it?.double }
 
-public fun MutableMetaProvider.long(key: Name? = null): ReadWriteProperty<Any?, Long?> =
+public fun MutableMetaProvider.long(key: Name? = null): MutableMetaDelegate<Long?> =
     value(key) { it?.long }
 
-public fun MutableMetaProvider.float(key: Name? = null): ReadWriteProperty<Any?, Float?> =
+public fun MutableMetaProvider.float(key: Name? = null): MutableMetaDelegate<Float?> =
     value(key) { it?.float }
 
 
 /* Safe number delegates*/
 
-public fun MutableMetaProvider.int(default: Int, key: Name? = null): ReadWriteProperty<Any?, Int> =
+public fun MutableMetaProvider.int(default: Int, key: Name? = null): MutableMetaDelegate<Int> =
     value(key) { it?.int ?: default }
 
-public fun MutableMetaProvider.double(default: Double, key: Name? = null): ReadWriteProperty<Any?, Double> =
+public fun MutableMetaProvider.double(default: Double, key: Name? = null): MutableMetaDelegate<Double> =
     value(key) { it?.double ?: default }
 
-public fun MutableMetaProvider.long(default: Long, key: Name? = null): ReadWriteProperty<Any?, Long> =
+public fun MutableMetaProvider.long(default: Long, key: Name? = null): MutableMetaDelegate<Long> =
     value(key) { it?.long ?: default }
 
-public fun MutableMetaProvider.float(default: Float, key: Name? = null): ReadWriteProperty<Any?, Float> =
+public fun MutableMetaProvider.float(default: Float, key: Name? = null): MutableMetaDelegate<Float> =
     value(key) { it?.float ?: default }
 
 
@@ -124,7 +218,7 @@ public fun MutableMetaProvider.float(default: Float, key: Name? = null): ReadWri
 public fun MutableMetaProvider.stringList(
     vararg default: String,
     key: Name? = null,
-): ReadWriteProperty<Any?, List<String>> = value(
+): MutableMetaDelegate<List<String>> = value(
     key,
     writer = { list -> list.map { str -> str.asValue() }.asValue() },
     reader = { it?.stringList ?: listOf(*default) },
@@ -132,7 +226,7 @@ public fun MutableMetaProvider.stringList(
 
 public fun MutableMetaProvider.stringList(
     key: Name? = null,
-): ReadWriteProperty<Any?, List<String>?> = value(
+): MutableMetaDelegate<List<String>?> = value(
     key,
     writer = { it -> it?.map { str -> str.asValue() }?.asValue() },
     reader = { it?.stringList },
@@ -141,29 +235,18 @@ public fun MutableMetaProvider.stringList(
 public fun MutableMetaProvider.numberList(
     vararg default: Number,
     key: Name? = null,
-): ReadWriteProperty<Any?, List<Number>> = value(
+): MutableMetaDelegate<List<Number>> = value(
     key,
     writer = { it.map { num -> num.asValue() }.asValue() },
     reader = { it?.list?.map { value -> value.numberOrNull ?: Double.NaN } ?: listOf(*default) },
 )
 
-/* A special delegate for double arrays */
-
-
-public fun MutableMetaProvider.doubleArray(
-    vararg default: Double,
-    key: Name? = null,
-): ReadWriteProperty<Any?, DoubleArray> = value(
-    key,
-    writer = { DoubleArrayValue(it) },
-    reader = { it?.doubleArray ?: doubleArrayOf(*default) },
-)
 
 public fun <T> MutableMetaProvider.listValue(
     key: Name? = null,
     writer: (T) -> Value = { Value.of(it) },
     reader: (Value) -> T,
-): ReadWriteProperty<Any?, List<T>?> = value(
+): MutableMetaDelegate<List<T>?> = value(
     key,
     writer = { it?.map(writer)?.asValue() },
     reader = { it?.list?.map(reader) }

dataforge-meta/src/commonMain/kotlin/space/kscience/dataforge/meta/MutableMetaView.kt

@@ -0,0 +1,47 @@
+package space.kscience.dataforge.meta
+
+import space.kscience.dataforge.names.Name
+import space.kscience.dataforge.names.NameToken
+import space.kscience.dataforge.names.parseAsName
+import space.kscience.dataforge.names.plus
+
+/**
+ * A [Meta] child proxy that creates required nodes on value write
+ */
+private class MutableMetaView(
+    val origin: MutableMeta,
+    val path: Name
+) : MutableMeta {
+
+    override val items: Map<NameToken, MutableMeta>
+        get() = origin[path]?.items ?: emptyMap()
+
+    override var value: Value?
+        get() = origin[path]?.value
+        set(value) {
+            origin[path] = value
+        }
+
+    override fun getOrCreate(name: Name): MutableMeta = MutableMetaView(origin, path + name)
+
+    override fun set(name: Name, node: Meta?) {
+        if (origin[path + name] == null && node?.isEmpty() == true) return
+        origin[path + name] = node
+    }
+
+    override fun equals(other: Any?): Boolean = Meta.equals(this, other as? Meta)
+
+    override fun hashCode(): Int = Meta.hashCode(this)
+
+    override fun toString(): String = Meta.toString(this)
+}
+
+/**
+ * Create a view of this [MutableMeta] node that creates child items only when their values are written.
+ *
+ * The difference between this method and regular [getOrCreate] is that [getOrCreate] always creates and attaches node
+ * even if it is empty.
+ */
+public fun MutableMeta.view(name: Name): MutableMeta = MutableMetaView(this, name)
+
+public fun MutableMeta.view(name: String): MutableMeta = view(name.parseAsName())

dataforge-meta/src/commonMain/kotlin/space/kscience/dataforge/meta/ObservableMeta.kt

@@ -1,8 +1,8 @@
 package space.kscience.dataforge.meta
 
+import kotlinx.serialization.Serializable
 import space.kscience.dataforge.misc.ThreadSafe
-import space.kscience.dataforge.names.*
+import space.kscience.dataforge.names.Name
-import kotlin.reflect.KProperty1
 
 
 internal data class MetaListener(
@@ -15,12 +15,15 @@ internal data class MetaListener(
  */
 public interface ObservableMeta : Meta {
     /**
-     * Add change listener to this meta. Owner is declared to be able to remove listeners later. Listener without owner could not be removed
+     * Add change listener to this meta. The Owner is declared to be able to remove listeners later.
+     * Listeners without an owner could be only removed all together.
+     *
+     * `this` object in the listener represents the current state of this meta. The name points to a changed node
      */
     public fun onChange(owner: Any?, callback: Meta.(name: Name) -> Unit)
 
     /**
-     * Remove all listeners belonging to given owner
+     * Remove all listeners belonging to the given [owner]. Passing null removes all listeners.
      */
     public fun removeListener(owner: Any?)
 
@@ -33,18 +36,10 @@ public interface ObservableMeta : Meta {
 /**
  * A [Meta] which is both observable and mutable
  */
+@Serializable(ObservableMutableMetaSerializer::class)
+@MetaBuilderMarker
 public interface ObservableMutableMeta : ObservableMeta, MutableMeta, MutableTypedMeta<ObservableMutableMeta> {
-    override fun getOrCreate(name: Name): ObservableMutableMeta
+    override val self: ObservableMutableMeta get() = this
-
-    override fun get(name: Name): ObservableMutableMeta? {
-        tailrec fun ObservableMutableMeta.find(name: Name): ObservableMutableMeta? = if (name.isEmpty()) {
-            this
-        } else {
-            items[name.firstOrNull()!!]?.find(name.cutFirst())
-        }
-
-        return find(name)
-    }
 }
 
 internal abstract class AbstractObservableMeta : ObservableMeta {
@@ -67,24 +62,4 @@ internal abstract class AbstractObservableMeta : ObservableMeta {
     override fun toString(): String = Meta.toString(this)
     override fun equals(other: Any?): Boolean = Meta.equals(this, other as? Meta)
     override fun hashCode(): Int = Meta.hashCode(this)
-}
-
-/**
- * Use the value of the property in a [callBack].
- * The callback is called once immediately after subscription to pass the initial value.
- *
- * Optional [owner] property is used for
- */
-public fun <S : Scheme, T> S.useProperty(
-    property: KProperty1<S, T>,
-    owner: Any? = null,
-    callBack: S.(T) -> Unit,
-) {
-    //Pass initial value.
-    callBack(property.get(this))
-    meta.onChange(owner) { name ->
-        if (name.startsWith(property.name.asName())) {
-            callBack(property.get(this@useProperty))
-        }
-    }
 }

dataforge-meta/src/commonMain/kotlin/space/kscience/dataforge/meta/ObservableMetaWrapper.kt

@@ -6,60 +6,74 @@ import space.kscience.dataforge.names.*
 
 /**
  * A class that takes [MutableMeta] provider and adds obsevability on top of that
+ *
+ * TODO rewrite to properly work with detached nodes
  */
 private class ObservableMetaWrapper(
     val root: MutableMeta,
-    val absoluteName: Name,
+    val nodeName: Name,
     val listeners: MutableSet<MetaListener>,
 ) : ObservableMutableMeta {
-    override val items: Map<NameToken, ObservableMutableMeta>
-        get() = root.items.keys.associateWith {
-            ObservableMetaWrapper(root, absoluteName + it, listeners)
-        }
 
-    override fun get(name: Name): ObservableMutableMeta? =
+    override val self get() = this
-        root.get(name)?.let { ObservableMetaWrapper(root, this.absoluteName + name, listeners) }
+
+    override val items: Map<NameToken, ObservableMutableMeta>
+        get() = root[nodeName]?.items?.keys?.associateWith {
+            ObservableMetaWrapper(root, nodeName + it, listeners)
+        } ?: emptyMap()
+
+    override fun get(name: Name): ObservableMutableMeta? = if (root[nodeName + name] == null) {
+        null
+    } else {
+        ObservableMetaWrapper(root, nodeName + name, listeners)
+    }
 
     @ThreadSafe
     override fun onChange(owner: Any?, callback: Meta.(name: Name) -> Unit) {
         listeners.add(
-            MetaListener(Pair(owner, absoluteName)) { name ->
+            MetaListener(Pair(owner, nodeName)) { fullName ->
-                if (name.startsWith(absoluteName)) {
+                if (fullName.startsWith(nodeName)) {
-                    (this[absoluteName] ?: Meta.EMPTY).callback(name.removeFirstOrNull(absoluteName)!!)
+                    root[nodeName]?.callback(fullName.removeFirstOrNull(nodeName)!!)
                 }
             }
         )
     }
 
     override fun removeListener(owner: Any?) {
-        listeners.removeAll { it.owner === Pair(owner, absoluteName) }
+        listeners.removeAll { it.owner === Pair(owner, nodeName) }
     }
 
     override fun invalidate(name: Name) {
-        listeners.forEach { it.callback(this, name) }
+        listeners.forEach { it.callback(this, nodeName + name) }
     }
 
     override var value: Value?
-        get() = root.value
+        get() = root[nodeName]?.value
         set(value) {
-            root.value = value
+            root.getOrCreate(nodeName).value = value
             invalidate(Name.EMPTY)
         }
 
     override fun getOrCreate(name: Name): ObservableMutableMeta =
-        ObservableMetaWrapper(root, this.absoluteName + name, listeners)
+        ObservableMetaWrapper(root, nodeName + name, listeners)
+
+    fun removeNode(name: Name): Meta? {
+        val oldMeta = get(name)
+        //remember to remove listener
+        oldMeta?.removeListener(this)
+
+        return oldMeta
+    }
 
     override fun set(name: Name, node: Meta?) {
-        val oldMeta = get(name)
+        val oldMeta = removeNode(name)
-        //don't forget to remove listener
+        root[nodeName + name] = node
-        oldMeta?.removeListener(this)
-        root.set(absoluteName + name, node)
         if (oldMeta != node) {
             invalidate(name)
         }
     }
 
-    override fun toMeta(): Meta = root[absoluteName]?.toMeta() ?: Meta.EMPTY
+    override fun toMeta(): Meta = root[nodeName]?.toMeta() ?: Meta.EMPTY
 
     override fun toString(): String = Meta.toString(this)
     override fun equals(other: Any?): Boolean = Meta.equals(this, other as? Meta)

dataforge-meta/src/commonMain/kotlin/space/kscience/dataforge/meta/Scheme.kt

@@ -7,36 +7,45 @@ import space.kscience.dataforge.meta.descriptors.validate
 import space.kscience.dataforge.misc.DFExperimental
 import space.kscience.dataforge.misc.ThreadSafe
 import space.kscience.dataforge.names.*
+import kotlin.properties.ReadWriteProperty
+import kotlin.reflect.KProperty
+import kotlin.reflect.KProperty1
 
 /**
- * A base for delegate-based or descriptor-based scheme. [Scheme] has an empty constructor to simplify usage from [Specification].
+ * A base for delegate-based or descriptor-based scheme. [Scheme] has an empty constructor to simplify usage from [MetaReader].
- * Default item provider and [MetaDescriptor] are optional
+ *
+ * @param prototype default values provided by this scheme
  */
-public open class Scheme : Described, MetaRepr, MutableMetaProvider, Configurable {
+public open class Scheme(
+    private var prototype: Meta? = null,
+    descriptor: MetaDescriptor? = null,
+) : Described, MetaRepr, MutableMetaProvider, Configurable {
 
     /**
-     * Meta to be mutated by this schme
+     * Meta to be mutated by this scheme
      */
-    private var targetMeta: MutableMeta = MutableMeta()
+    internal var target: MutableMeta = MutableMeta()
 
     /**
-     * Default values provided by this scheme
+     * A descriptor of this scheme
      */
-    private var defaultMeta: Meta? = null
+    final override var descriptor: MetaDescriptor? = descriptor
+        private set
+
 
     final override val meta: ObservableMutableMeta = SchemeMeta(Name.EMPTY)
 
-    final override var descriptor: MetaDescriptor? = null
+    /**
-        internal set
+     * This method must be called before the scheme could be used
-
+     */
-    internal fun wrap(
+    internal fun initialize(
-        newMeta: MutableMeta,
+        target: MutableMeta,
-        preserveDefault: Boolean = false,
+        prototype: Meta,
+        descriptor: MetaDescriptor?,
     ) {
-        if (preserveDefault) {
+        this.target = target
-            defaultMeta = targetMeta.seal()
+        this.prototype = prototype
-        }
+        this.descriptor = descriptor
-        targetMeta = newMeta
     }
 
     /**
@@ -47,11 +56,11 @@ public open class Scheme : Described, MetaRepr, MutableMetaProvider, Configurabl
         return descriptor?.validate(meta) ?: true
     }
 
-    override fun get(name: Name): MutableMeta? = meta.get(name)
+    override fun get(name: Name): MutableMeta? = meta[name]
 
     override fun set(name: Name, node: Meta?) {
         if (validate(name, meta)) {
-            meta.set(name, node)
+            meta[name] = node
         } else {
             error("Validation failed for node $node at $name")
         }
@@ -68,14 +77,19 @@ public open class Scheme : Described, MetaRepr, MutableMetaProvider, Configurabl
 
     private val listeners: MutableList<MetaListener> = mutableListOf()
 
+    override fun toString(): String = meta.toString()
+
     private inner class SchemeMeta(val pathName: Name) : ObservableMutableMeta {
+
+        override val self get() = this
+
         override var value: Value?
-            get() = targetMeta[pathName]?.value
+            get() = target[pathName]?.value
-                ?: defaultMeta?.get(pathName)?.value
+                ?: prototype?.get(pathName)?.value
                 ?: descriptor?.get(pathName)?.defaultValue
             set(value) {
-                val oldValue = targetMeta[pathName]?.value
+                val oldValue = target[pathName]?.value
-                targetMeta[pathName] = value
+                target[pathName] = value
                 if (oldValue != value) {
                     invalidate(Name.EMPTY)
                 }
@@ -83,8 +97,8 @@ public open class Scheme : Described, MetaRepr, MutableMetaProvider, Configurabl
 
         override val items: Map<NameToken, ObservableMutableMeta>
             get() {
-                val targetKeys = targetMeta[pathName]?.items?.keys ?: emptySet()
+                val targetKeys = target[pathName]?.items?.keys ?: emptySet()
-                val defaultKeys = defaultMeta?.get(pathName)?.items?.keys ?: emptySet()
+                val defaultKeys = prototype?.get(pathName)?.items?.keys ?: emptySet()
                 return (targetKeys + defaultKeys).associateWith { SchemeMeta(pathName + it) }
             }
 
@@ -111,7 +125,7 @@ public open class Scheme : Described, MetaRepr, MutableMetaProvider, Configurabl
         override fun hashCode(): Int = Meta.hashCode(this)
 
         override fun set(name: Name, node: Meta?) {
-            targetMeta.set(name, node)
+            target[pathName + name] = node
             invalidate(name)
         }
 
@@ -119,7 +133,6 @@ public open class Scheme : Described, MetaRepr, MutableMetaProvider, Configurabl
 
         @DFExperimental
         override fun attach(name: Name, node: ObservableMutableMeta) {
-            //TODO implement zero-copy attachment
             set(name, node)
             node.onChange(this) { changeName ->
                 set(name + changeName, this[changeName])
@@ -131,10 +144,11 @@ public open class Scheme : Described, MetaRepr, MutableMetaProvider, Configurabl
 
 /**
  * Relocate scheme target onto given [MutableMeta]. Old provider does not get updates anymore.
- * Current state of the scheme used as a default.
+ * The Current state of the scheme that os used as a default.
  */
+@DFExperimental
 public fun <T : Scheme> T.retarget(provider: MutableMeta): T = apply {
-    wrap(provider, true)
+    initialize(provider, meta.seal(), descriptor)
 }
 
 /**
@@ -151,26 +165,151 @@ public inline fun <T : Scheme> T.copy(spec: SchemeSpec<T>, block: T.() -> Unit =
 /**
  * A specification for simplified generation of wrappers
  */
-public open class SchemeSpec<out T : Scheme>(
+public open class SchemeSpec<T : Scheme>(
     private val builder: () -> T,
-) : Specification<T> {
+) : MetaConverter<T> {
 
-    override fun read(source: Meta): T = builder().also {
-        it.wrap(MutableMeta().withDefault(source))
-    }
-
-    override fun write(target: MutableMeta): T = empty().also {
-        it.wrap(target)
-    }
-
-    //TODO Generate descriptor from Scheme class
     override val descriptor: MetaDescriptor? get() = null
 
-    override fun empty(): T = builder().also {
+    override fun readOrNull(source: Meta): T = builder().also {
-        it.descriptor = descriptor
+        it.initialize(MutableMeta(), source, descriptor)
     }
 
-    @Suppress("OVERRIDE_BY_INLINE")
+    /**
-    final override inline operator fun invoke(action: T.() -> Unit): T = empty().apply(action)
+     * Write changes made to the [Scheme] to target [MutableMeta]. If the empty [Scheme] contains any data it is copied to the target.
+     */
+    public fun write(target: MutableMeta): T = empty().also {
+        target.update(it.meta)
+        it.initialize(target, Meta.EMPTY, descriptor)
+    }
 
+    /**
+     * Generate a blank object. The object could contain some elements if they are defined in a constructor
+     */
+    public fun empty(): T = builder().also {
+        it.initialize(MutableMeta(), it.target, descriptor)
+    }
+
+    override fun convert(obj: T): Meta = obj.meta
+
+    /**
+     * A convenience method to use specifications in builders
+     */
+    public inline operator fun invoke(action: T.() -> Unit): T = empty().apply(action)
+
+}
+
+
+/**
+ * Update a [MutableMeta] using given specification
+ */
+public fun <T : Scheme> MutableMeta.updateWith(
+    spec: SchemeSpec<T>,
+    action: T.() -> Unit,
+): T = spec.write(this).apply(action)
+
+
+/**
+ * Update configuration using given specification
+ */
+public fun <T : Scheme> Configurable.updateWith(
+    spec: SchemeSpec<T>,
+    action: T.() -> Unit,
+): T = spec.write(meta).apply(action)
+
+
+/**
+ * A delegate that uses a [MetaReader] to wrap a child of this provider
+ */
+public fun <T : Scheme> MutableMetaProvider.scheme(
+    spec: SchemeSpec<T>,
+    key: Name? = null,
+): ReadWriteProperty<Any?, T> = object : ReadWriteProperty<Any?, T> {
+    override fun getValue(thisRef: Any?, property: KProperty<*>): T {
+        val name = key ?: property.name.asName()
+        val node = get(name) ?: MutableMeta().also { set(name, it) }
+        return spec.write(node)
+    }
+
+    override fun setValue(thisRef: Any?, property: KProperty<*>, value: T) {
+        val name = key ?: property.name.asName()
+        set(name, value.toMeta())
+    }
+}
+
+public fun <T : Scheme> Scheme.scheme(
+    spec: SchemeSpec<T>,
+    key: Name? = null,
+): ReadWriteProperty<Any?, T> = meta.scheme(spec, key)
+
+/**
+ * A delegate that uses a [MetaReader] to wrap a child of this provider.
+ * Returns null if meta with given name does not exist.
+ */
+public fun <T : Scheme> MutableMeta.schemeOrNull(
+    spec: SchemeSpec<T>,
+    key: Name? = null,
+): ReadWriteProperty<Any?, T?> = object : ReadWriteProperty<Any?, T?> {
+    override fun getValue(thisRef: Any?, property: KProperty<*>): T? {
+        val name = key ?: property.name.asName()
+        return if (get(name) == null) null else spec.write(getOrCreate(name))
+    }
+
+    override fun setValue(thisRef: Any?, property: KProperty<*>, value: T?) {
+        val name = key ?: property.name.asName()
+        if (value == null) remove(name)
+        else set(name, value.toMeta())
+    }
+}
+
+public fun <T : Scheme> Scheme.schemeOrNull(
+    spec: SchemeSpec<T>,
+    key: Name? = null,
+): ReadWriteProperty<Any?, T?> = meta.schemeOrNull(spec, key)
+
+/**
+ * A delegate that uses a [MetaReader] to wrap a list of child providers.
+ * If children are mutable, the changes in list elements are reflected on them.
+ * The list is a snapshot of children state, so change in structure is not reflected on its composition.
+ */
+public fun <T : Scheme> MutableMeta.listOfScheme(
+    spec: SchemeSpec<T>,
+    key: Name? = null,
+): ReadWriteProperty<Any?, List<T>> = object : ReadWriteProperty<Any?, List<T>> {
+    override fun getValue(thisRef: Any?, property: KProperty<*>): List<T> {
+        val name = key ?: property.name.asName()
+        return getIndexedList(name).map { spec.write(it as MutableMeta) }
+    }
+
+    override fun setValue(thisRef: Any?, property: KProperty<*>, value: List<T>) {
+        val name = key ?: property.name.asName()
+        setIndexed(name, value.map { it.toMeta() })
+    }
+}
+
+
+public fun <T : Scheme> Scheme.listOfScheme(
+    spec: SchemeSpec<T>,
+    key: Name? = null,
+): ReadWriteProperty<Any?, List<T>> = meta.listOfScheme(spec, key)
+
+
+/**
+ * Use the value of the property in a [callBack].
+ * The callback is called once immediately after subscription to pass the initial value.
+ *
+ * Optional [owner] property is used for
+ */
+public fun <S : Scheme, T> S.useProperty(
+    property: KProperty1<S, T>,
+    owner: Any? = null,
+    callBack: S.(T) -> Unit,
+) {
+    //Pass initial value.
+    callBack(property.get(this))
+    meta.onChange(owner) { name ->
+        if (name.startsWith(property.name.asName())) {
+            callBack(property.get(this@useProperty))
+        }
+    }
 }

dataforge-meta/src/commonMain/kotlin/space/kscience/dataforge/meta/SealedMeta.kt

@@ -13,6 +13,9 @@ public class SealedMeta(
     override val value: Value?,
     override val items: Map<NameToken, SealedMeta>,
 ) : TypedMeta<SealedMeta> {
+
+    override val self: SealedMeta get() = this
+
     override fun toString(): String = Meta.toString(this)
     override fun equals(other: Any?): Boolean = Meta.equals(this, other as? Meta)
 
@@ -76,6 +79,8 @@ internal class MetaBuilder(
 
 
     override fun set(name: Name, node: Meta?) {
+        //skip setting if value has not changed
+        if(node == get(name)) return
         when (name.length) {
             0 -> error("Can't set a meta with empty name")
             1 -> {
@@ -89,7 +94,7 @@ internal class MetaBuilder(
             }
 
             else -> {
-                getOrCreate(name.first().asName()).set(name.cutFirst(), node)
+                getOrCreate(name.first().asName())[name.cutFirst()] = node
             }
         }
     }
@@ -101,11 +106,6 @@ internal class MetaBuilder(
     override fun hashCode(): Int = Meta.hashCode(this)
 }
 
-/**
- * Create a read-only meta.
- */
-public inline fun Meta(builder: MutableMeta.() -> Unit): Meta =
-    MetaBuilder().apply(builder).seal()
 
 /**
  * Create an immutable meta.
@@ -113,6 +113,11 @@ public inline fun Meta(builder: MutableMeta.() -> Unit): Meta =
 public inline fun SealedMeta(builder: MutableMeta.() -> Unit): SealedMeta =
     MetaBuilder().apply(builder).seal()
 
+/**
+ * Create a read-only meta.
+ */
+public inline fun Meta(builder: MutableMeta.() -> Unit): Meta = SealedMeta(builder)
+
 /**
  * Create an empty meta mutable meta.
  */

dataforge-meta/src/commonMain/kotlin/space/kscience/dataforge/meta/Specification.kt

@@ -1,130 +0,0 @@
-package space.kscience.dataforge.meta
-
-import space.kscience.dataforge.meta.descriptors.Described
-import space.kscience.dataforge.misc.DFExperimental
-import space.kscience.dataforge.names.Name
-import space.kscience.dataforge.names.asName
-import kotlin.properties.ReadWriteProperty
-import kotlin.reflect.KProperty
-
-public interface ReadOnlySpecification<out T : Any>: Described {
-
-    /**
-     * Read generic read-only meta with this [Specification] producing instance of desired type.
-     * The source is not mutated even if it is in theory mutable
-     */
-    public fun read(source: Meta): T
-
-    /**
-     * Generate an empty object
-     */
-    public fun empty(): T
-
-    /**
-     * A convenience method to use specifications in builders
-     */
-    public operator fun invoke(action: T.() -> Unit): T = empty().apply(action)
-}
-
-
-/**
- * Allows to apply custom configuration in a type safe way to simple untyped configuration.
- * By convention [Scheme] companion should inherit this class
- *
- */
-public interface Specification<out T : Any> : ReadOnlySpecification<T> {
-    /**
-     * Wrap [MutableMeta], using it as inner storage (changes to [Specification] are reflected on [MutableMeta]
-     */
-    public fun write(target: MutableMeta): T
-}
-
-/**
- * Update a [MutableMeta] using given specification
- */
-public fun <T : Any> MutableMeta.updateWith(
-    spec: Specification<T>,
-    action: T.() -> Unit,
-): T = spec.write(this).apply(action)
-
-
-/**
- * Update configuration using given specification
- */
-public fun <T : Any> Configurable.updateWith(
-    spec: Specification<T>,
-    action: T.() -> Unit,
-): T = spec.write(meta).apply(action)
-
-//
-//public fun  <M : MutableTypedMeta<M>> MutableMeta.withSpec(spec: Specification<M>): M? =
-//    spec.write(it)
-
-/**
- * A delegate that uses a [Specification] to wrap a child of this provider
- */
-public fun <T : Scheme> MutableMeta.spec(
-    spec: Specification<T>,
-    key: Name? = null,
-): ReadWriteProperty<Any?, T> = object : ReadWriteProperty<Any?, T> {
-    override fun getValue(thisRef: Any?, property: KProperty<*>): T {
-        val name = key ?: property.name.asName()
-        return spec.write(getOrCreate(name))
-    }
-
-    override fun setValue(thisRef: Any?, property: KProperty<*>, value: T) {
-        val name = key ?: property.name.asName()
-        set(name, value.toMeta())
-    }
-}
-
-public fun <T : Scheme> Scheme.spec(
-    spec: Specification<T>,
-    key: Name? = null,
-): ReadWriteProperty<Any?, T> = meta.spec(spec, key)
-
-/**
- * A delegate that uses a [Specification] to wrap a child of this provider.
- * Returns null if meta with given name does not exist.
- */
-public fun <T : Scheme> MutableMeta.specOrNull(
-    spec: Specification<T>,
-    key: Name? = null,
-): ReadWriteProperty<Any?, T?> = object : ReadWriteProperty<Any?, T?> {
-    override fun getValue(thisRef: Any?, property: KProperty<*>): T? {
-        val name = key ?: property.name.asName()
-        return if (get(name) == null) null else spec.write(getOrCreate(name))
-    }
-
-    override fun setValue(thisRef: Any?, property: KProperty<*>, value: T?) {
-        val name = key ?: property.name.asName()
-        if (value == null) remove(name)
-        else set(name, value.toMeta())
-    }
-}
-
-public fun <T : Scheme> Scheme.specOrNull(
-    spec: Specification<T>,
-    key: Name? = null,
-): ReadWriteProperty<Any?, T?> = meta.specOrNull(spec, key)
-
-/**
- * A delegate that uses a [Specification] to wrap a list of child providers.
- * If children are mutable, the changes in list elements are reflected on them.
- * The list is a snapshot of children state, so change in structure is not reflected on its composition.
- */
-@DFExperimental
-public fun <T : Scheme> MutableMeta.listOfSpec(
-    spec: Specification<T>,
-    key: Name? = null,
-): ReadWriteProperty<Any?, List<T>> = object : ReadWriteProperty<Any?, List<T>> {
-    override fun getValue(thisRef: Any?, property: KProperty<*>): List<T> {
-        val name = key ?: property.name.asName()
-        return getIndexed(name).values.map { spec.write(it as MutableMeta) }
-    }
-
-    override fun setValue(thisRef: Any?, property: KProperty<*>, value: List<T>) {
-        val name = key ?: property.name.asName()
-        setIndexed(name, value.map { it.toMeta() })
-    }
-}

dataforge-meta/src/commonMain/kotlin/space/kscience/dataforge/meta/Value.kt

@@ -256,8 +256,6 @@ public fun ShortArray.asValue(): Value = if (isEmpty()) Null else ListValue(map
 
 public fun FloatArray.asValue(): Value = if (isEmpty()) Null else ListValue(map { NumberValue(it) })
 
-public fun ByteArray.asValue(): Value = if (isEmpty()) Null else ListValue(map { NumberValue(it) })
-
 public fun <E : Enum<E>> E.asValue(): Value = EnumValue(this)
 
 

dataforge-meta/src/commonMain/kotlin/space/kscience/dataforge/meta/ValueSerializer.kt

@@ -8,6 +8,9 @@ import kotlinx.serialization.descriptors.element
 import kotlinx.serialization.encoding.Decoder
 import kotlinx.serialization.encoding.Encoder
 
+/**
+ * A serializer for [Value]
+ */
 public object ValueSerializer : KSerializer<Value> {
     private val listSerializer by lazy { ListSerializer(ValueSerializer) }
 

dataforge-meta/src/commonMain/kotlin/space/kscience/dataforge/meta/descriptors/MetaDescriptor.kt

@@ -7,6 +7,7 @@ import space.kscience.dataforge.names.*
 /**
  * Restrictions on value in the node
  */
+@Serializable
 public enum class ValueRestriction {
     /**
      * No restrictions
@@ -27,7 +28,7 @@ public enum class ValueRestriction {
 /**
  * The descriptor for a meta
  * @param description description text
- * @param children child descriptors for this node
+ * @param nodes child descriptors for this node
  * @param multiple True if same name siblings with this name are allowed
  * @param valueRestriction The requirements for node content
  * @param valueTypes list of allowed types for [Meta.value], null if all values are allowed.
@@ -39,7 +40,7 @@ public enum class ValueRestriction {
 @Serializable
 public data class MetaDescriptor(
     public val description: String? = null,
-    public val children: Map<String, MetaDescriptor> = emptyMap(),
+    public val nodes: Map<String, MetaDescriptor> = emptyMap(),
     public val multiple: Boolean = false,
     public val valueRestriction: ValueRestriction = ValueRestriction.NONE,
     public val valueTypes: List<ValueType>? = null,
@@ -47,6 +48,9 @@ public data class MetaDescriptor(
     public val defaultValue: Value? = null,
     public val attributes: Meta = Meta.EMPTY,
 ) {
+    @Deprecated("Replace by nodes", ReplaceWith("nodes"))
+    public val children: Map<String, MetaDescriptor> get() = nodes
+
     /**
      * A node constructed of default values for this descriptor and its children
      */
@@ -55,7 +59,7 @@ public data class MetaDescriptor(
             defaultValue?.let { defaultValue ->
                 this.value = defaultValue
             }
-            children.forEach { (key, descriptor) ->
+            nodes.forEach { (key, descriptor) ->
                 set(key, descriptor.defaultNode)
             }
         }
@@ -67,13 +71,13 @@ public data class MetaDescriptor(
     }
 }
 
-public val MetaDescriptor.required: Boolean get() = valueRestriction == ValueRestriction.REQUIRED || children.values.any { required }
+public val MetaDescriptor.required: Boolean get() = valueRestriction == ValueRestriction.REQUIRED || nodes.values.any { required }
 
 public val MetaDescriptor.allowedValues: List<Value>? get() = attributes[MetaDescriptor.ALLOWED_VALUES_KEY]?.value?.list
 
 public operator fun MetaDescriptor.get(name: Name): MetaDescriptor? = when (name.length) {
     0 -> this
-    1 -> children[name.firstOrNull()!!.toString()]
+    1 -> nodes[name.firstOrNull()!!.toString()]
     else -> get(name.firstOrNull()!!.asName())?.get(name.cutFirst())
 }
 
@@ -95,7 +99,7 @@ public fun MetaDescriptor.validate(item: Meta?): Boolean {
     if (item == null) return !required
     if (!validate(item.value)) return false
 
-    children.forEach { (key, childDescriptor) ->
+    nodes.forEach { (key, childDescriptor) ->
         if (!childDescriptor.validate(item[key])) return false
     }
     return true