Fix type capture for raw html parser

.gitignore

@@ -1,5 +1,6 @@
 .gradle/
 build/
+.kotlin/
 .idea/
 /logs/
 

README.md

@@ -1,2 +1,100 @@
-# snark
-Scientific Notation And Representation in Kotlin
+# SNARK
+
+In Lewis Carroll "The hunting of the Snark", the Snark itself is something everybody want to get, but nobody know what it is. It is the same in case of this project, but it also has narrower scope. SNARK could be read as "Scientific Notation And Research works in Kotlin" because it could be used for automatic creation of research papers. But it has other purposes as well.
+
+To sum it up, **SNARK is an automated data transformation tool with the main focus on document and web page generation**. It is based on [DataForge framework](https://github.com/SciProgCentre/dataforge-core).
+
+SNARK **is not a typesetting system** itself, but it could utilize typesetting systems such as Markdown, Latex or Typst to do data transformations.
+
+## Concepts
+
+The SNARK process it the transformation of a data tree. Initial data could include texts, images, static binary or textual data or even active external data subscriptions. The result is usually a tree of documents or a directly served web-site.
+
+**Data** is any kind of content, generated lazily with additional metadata (DataForge Meta).
+
+## Using DataForge context
+
+DataForge module management is based on **Contexts** and **Plugins**. Context is used both as dependency injection system, lifecycle object and API discoverability root for all executions. To use some subsystem, one needs to:
+
+* Create a Context with a Plugin like this:
+
+```kotlin
+Context("Optional context name"){
+    plugin(SnarkHtml) 
+    // Here SnarkHtml is a Plugin factory declared as a companion object to a Plugin itself
+}
+```
+
+* Get the loaded plugin instance via `val snarkHtml = context.request(SnarkHtml)`
+
+* Use plugin like
+
+```kotlin
+val siteData = snarkHtml.readSiteData(context) {
+    directory(snark.io, Name.EMPTY, dataDirectory)
+}
+
+```
+
+## SNARK-html
+
+SNARK-HTML module defines tools to work with HTML output format. The API root for it is `SnarkHtml` plugin. Its primary function (`parse` action) is to parse raw binary DataTree with objects specific for HTML rendering, assets and metadata. It uses `SnarkReader` and more specifically `SnarkHtmlReader` to parse binary data into formats like `Meta` and `PageFragment`. If `parse` could not recognize the format of the input, it leaves it as (lazy) binary.
+
+### Preprocessing and postprocessing
+
+Snark uses DataForge data tree transformation ideology so there could be any number of data transformation steps both before parsing and after parsing, but there is a key difference: before parsing we work with binaries that could be transformed directly (yet lazily because this is how DataForge works), after parsing we have not a hard data, but a rendering function that could only be transformed by wrapping it in another function (which could be complicated). The raw data transformation before parsing is called preprocessing. It could include both raw binary transformation and metadata transformation. The postprocessing is usually done inside the rendering function produced by parser or created directly from code.
+
+The interface for `PageFragment` looks like this: 
+
+```kotlin
+public fun interface PageFragment {
+    context(PageContextWithData, FlowContent) public fun renderFragment()
+}
+```
+
+It takes a reference to parsed data tree and rendering context of the page as well as HTML mounting root and provides action to render HTML. The reason for such complication is that some things are not known before the actual page rendering happens. For example, absolute links in HTML could be resolved only when the page is rendered on specific REST request that contain information about host and port. Another example is providing automatic counters for chapters, formulas and images in document rendering. The order is not known until all fragments are placed in correct order.
+
+Postprocessors are functions that transform fragments of HTML wrapped in them according to data tree and page rendering context.
+
+Other details on HTML rendering could be found in [snark-html](./snark-html) module
+
+
+
+### [examples](examples)
+>
+> **Maturity**: EXPERIMENTAL
+
+### [snark-core](snark-core)
+>
+> **Maturity**: EXPERIMENTAL
+
+### [snark-gradle-plugin](snark-gradle-plugin)
+>
+> **Maturity**: EXPERIMENTAL
+
+### [snark-html](snark-html)
+>
+> **Maturity**: EXPERIMENTAL
+>
+> **Features:**
+> - [data](snark-html/#) : Data-based processing. Instead of traditional layout-based
+> - [layouts](snark-html/#) : Use custom layouts to represent a data tree
+> - [parsers](snark-html/#) : Add custom file formats and parsers using DataForge dependency injection
+> - [preprocessor](snark-html/#) : Preprocessing text files using templates
+> - [metadata](snark-html/#) : Trademark DataForge metadata layering and transformations
+> - [dynamic](snark-html/#) : Generating dynamic site using KTor server
+> - [static](snark-html/#) : Generating static site
+
+
+### [snark-ktor](snark-ktor)
+>
+> **Maturity**: EXPERIMENTAL
+
+### [snark-pandoc](snark-pandoc)
+>
+> **Maturity**: EXPERIMENTAL
+
+### [examples/document](examples/document)
+>
+> **Maturity**: EXPERIMENTAL
+

build.gradle.kts

@@ -22,6 +22,10 @@ ksciencePublish {
         useApache2Licence()
         useSPCTeam()
     }
-    repository("spc","https://maven.sciprog.center/kscience")
+    repository("spc", "https://maven.sciprog.center/kscience")
 //    sonatype()
+}
+
+readme {
+    this.useDefaultReadmeTemplate
 }

docs/README-TEMPLATE.md

@@ -0,0 +1,71 @@
+# SNARK
+
+In Lewis Carroll "The hunting of the Snark", the Snark itself is something everybody want to get, but nobody know what it is. It is the same in case of this project, but it also has narrower scope. SNARK could be read as "Scientific Notation And Research works in Kotlin" because it could be used for automatic creation of research papers. But it has other purposes as well.
+
+To sum it up, **SNARK is an automated data transformation tool with the main focus on document and web page generation**. It is based on [DataForge framework](https://github.com/SciProgCentre/dataforge-core).
+
+SNARK **is not a typesetting system** itself, but it could utilize typesetting systems such as Markdown, Latex or Typst to do data transformations.
+
+## Concepts
+
+The SNARK process it the transformation of a data tree. Initial data could include texts, images, static binary or textual data or even active external data subscriptions. The result is usually a tree of documents or a directly served web-site.
+
+**Data** is any kind of content, generated lazily with additional metadata (DataForge Meta).
+
+## Using DataForge context
+
+DataForge module management is based on **Contexts** and **Plugins**. Context is used both as dependency injection system, lifecycle object and API discoverability root for all executions. To use some subsystem, one needs to:
+
+* Create a Context with a Plugin like this:
+
+```kotlin
+Context("Optional context name"){
+    plugin(SnarkHtml) 
+    // Here SnarkHtml is a Plugin factory declared as a companion object to a Plugin itself
+}
+```
+
+* Get the loaded plugin instance via `val snarkHtml = context.request(SnarkHtml)`
+
+* Use plugin like
+
+```kotlin
+val siteData = snarkHtml.readSiteData(context) {
+    directory(snark.io, Name.EMPTY, dataDirectory)
+}
+
+```
+
+## SNARK-html
+
+SNARK-HTML module defines tools to work with HTML output format. The API root for it is `SnarkHtml` plugin. Its primary function (`parse` action) is to parse raw binary DataTree with objects specific for HTML rendering, assets and metadata. It uses `SnarkReader` and more specifically `SnarkHtmlReader` to parse binary data into formats like `Meta` and `PageFragment`. If `parse` could not recognize the format of the input, it leaves it as (lazy) binary.
+
+### Preprocessing and postprocessing
+
+Snark uses DataForge data tree transformation ideology so there could be any number of data transformation steps both before parsing and after parsing, but there is a key difference: before parsing we work with binaries that could be transformed directly (yet lazily because this is how DataForge works), after parsing we have not a hard data, but a rendering function that could only be transformed by wrapping it in another function (which could be complicated). The raw data transformation before parsing is called preprocessing. It could include both raw binary transformation and metadata transformation. The postprocessing is usually done inside the rendering function produced by parser or created directly from code.
+
+The interface for `PageFragment` looks like this: 
+
+```kotlin
+public fun interface PageFragment {
+    context(PageContextWithData, FlowContent) public fun renderFragment()
+}
+```
+
+It takes a reference to parsed data tree and rendering context of the page as well as HTML mounting root and provides action to render HTML. The reason for such complication is that some things are not known before the actual page rendering happens. For example, absolute links in HTML could be resolved only when the page is rendered on specific REST request that contain information about host and port. Another example is providing automatic counters for chapters, formulas and images in document rendering. The order is not known until all fragments are placed in correct order.
+
+Postprocessors are functions that transform fragments of HTML wrapped in them according to data tree and page rendering context.
+
+Other details on HTML rendering could be found in [snark-html](./snark-html) module
+
+## Examples
+
+### Scientific document builder
+
+The idea of [the project](examples/document) is to produce a tree of scientific documents or papers. It does that in following steps:
+
+1. Read data tree from `data` directory (data path could be overridden by either ktor configuration or manually).
+2. Search all directories for a files called `document.yaml` or any other format that could be treated as value-tree (for example `document.json`). Use that file as a document descriptor that defines linear document structure.
+3. 
+
+${modules}

examples/README.md

@@ -0,0 +1,4 @@
+# Module examples
+
+
+

examples/document/README.md

@@ -0,0 +1,4 @@
+# Module document
+
+
+

examples/document/build.gradle.kts

@@ -1,33 +1,33 @@
 plugins {
     id("space.kscience.gradle.mpp")
-    application
+    alias(spclibs.plugins.ktor)
 }
 
-application {
-    mainClass.set("Mainkt")
-
-    val isDevelopment: Boolean = project.ext.has("development")
-    applicationDefaultJvmArgs = listOf("-Dio.ktor.development=$isDevelopment", "-Xmx200M")
-}
-
-val snarkVersion: String by extra
-val ktorVersion = space.kscience.gradle.KScienceVersions.ktorVersion
-
 kscience {
-    jvm()
+    jvm{
+        withJava()
+    }
     useContextReceivers()
+    useKtor()
 
     jvmMain {
         implementation(projects.snarkKtor)
-        implementation("io.ktor:ktor-server-cio:$ktorVersion")
+        implementation("io.ktor:ktor-server-cio")
         implementation(spclibs.logback.classic)
     }
 
-    jvmTest{
-        implementation("io.ktor:ktor-server-tests:$ktorVersion")
+    jvmTest {
+        implementation("io.ktor:ktor-server-tests")
     }
 }
 
 kotlin {
     explicitApi = org.jetbrains.kotlin.gradle.dsl.ExplicitApiMode.Disabled
+}
+
+application {
+    mainClass.set("center.sciprog.snark.documents.MainKt")
+
+    val isDevelopment: Boolean = project.ext.has("development")
+    applicationDefaultJvmArgs = listOf("-Dio.ktor.development=$isDevelopment", "-Xmx200M")
 }

examples/document/data/loremIpsum/chapter1.md

@@ -12,6 +12,10 @@ ${documentMeta.get('metaValue')}
 
 Curabitur hendrerit hendrerit rutrum. Nullam elementum libero a nisi viverra aliquet. Sed ut urna a sem bibendum dictum. Cras non elit sit amet ex ultrices iaculis. Fusce lobortis lacinia fermentum. Fusce in metus id massa mollis consequat. Quisque non dolor quis orci gravida vulputate. Vivamus sed pellentesque orci. Sed aliquet malesuada rhoncus. Mauris id aliquet lorem.
 
+Paragraph
+
+$  \int_a^b {f(x)} = const $
+
 ### Section ${section(2)}
 
 Maecenas at iaculis ipsum. Praesent maximus tristique magna eu faucibus. In tincidunt elementum pharetra. Nam scelerisque eros mattis, suscipit odio sit amet, efficitur mi. Etiam eleifend pulvinar erat a aliquet. Cras pellentesque tincidunt mi eget scelerisque. Proin eget ipsum a velit lobortis commodo. Nulla facilisi. Donec id pretium leo. Ut nec tortor sapien. Praesent vehicula dolor ut laoreet commodo. Pellentesque convallis, sapien et placerat luctus, tortor magna sodales sem, non tristique eros sem vel ipsum. Nulla vulputate accumsan nulla. Duis tempor, mi nec pharetra suscipit, sem odio sagittis mi, ut dignissim odio erat a dolor.

examples/document/data/loremIpsum/document.yaml

@@ -2,7 +2,7 @@ route: lorem.ipsum
 title: Lorem Ipsum
 authors:
   - name: Alexander Nozik
-    affiliation: MIPT
+    affiliation: SPC
 fragments:
   - type: image
     ref: SPC-logo.png

examples/document/data/simple/body.md

@@ -0,0 +1,2 @@
+
+This is a document body for a simple document

examples/document/data/simple/document.json

@@ -0,0 +1,13 @@
+{
+  "title": "A simple document",
+  "fragments": [
+    {
+      "type": "data",
+      "name": "body"
+    },
+    {
+      "type": "data",
+      "name": "footer"
+    }
+  ]
+}

examples/document/data/simple/footer.html

@@ -0,0 +1,4 @@
+
+<p>
+    <strong>This is HTML footer</strong>
+</p>

examples/document/src/jvmMain/kotlin/main.kt

@@ -1,15 +1,24 @@
+package center.sciprog.snark.documents
+
 import io.ktor.server.application.Application
+import io.ktor.server.application.call
 import io.ktor.server.cio.CIO
 import io.ktor.server.engine.embeddedServer
+import io.ktor.server.response.respondRedirect
+import io.ktor.server.routing.get
+import io.ktor.server.routing.routing
 import kotlinx.html.ScriptCrossorigin
 import kotlinx.html.link
 import kotlinx.html.script
+import kotlinx.html.unsafe
 import space.kscience.snark.html.document.allDocuments
+import space.kscience.snark.ktor.snarkApplication
 
 @Suppress("unused")
 fun Application.renderAllDocuments() = snarkApplication {
     allDocuments(
         headers = {
+            //add katex headers
             link {
                 rel = "stylesheet"
                 href = "https://cdn.jsdelivr.net/npm/katex@0.16.10/dist/katex.min.css"
@@ -29,13 +38,34 @@ fun Application.renderAllDocuments() = snarkApplication {
                 crossorigin = ScriptCrossorigin.anonymous
                 attributes["onload"] = "renderMathInElement(document.body);"
             }
+            // Auto-render latex expressions with katex
+            script {
+                unsafe {
+                    +"""
+                        document.addEventListener("DOMContentLoaded", function() {
+                            renderMathInElement(document.body, {
+                                delimiters: [
+                                    {left: '$$', right: '$$', display: true},
+                                    {left: '$', right: '$', display: false},
+                                ],
+                                throwOnError : false
+                            });
+                        });
+
+                    """.trimIndent()
+                }
+            }
         }
     )
+
+    routing {
+        get("/"){
+            call.respondRedirect("lorem/ipsum")
+        }
+    }
 }
 
 
 fun main() {
-    embeddedServer(CIO) {
-        renderAllDocuments()
-    }.start(true)
+    embeddedServer(CIO, module = Application::renderAllDocuments).start(true)
 }

examples/document/src/jvmMain/kotlin/snarkApplication.kt

@@ -1,49 +0,0 @@
-import io.ktor.server.application.Application
-import io.ktor.server.application.log
-import io.ktor.server.http.content.staticResources
-import io.ktor.server.routing.routing
-import space.kscience.dataforge.context.Context
-import space.kscience.dataforge.context.ContextBuilder
-import space.kscience.dataforge.context.request
-import space.kscience.dataforge.data.forEach
-import space.kscience.dataforge.names.Name
-import space.kscience.dataforge.workspace.FileData
-import space.kscience.dataforge.workspace.directory
-import space.kscience.snark.html.HtmlSite
-import space.kscience.snark.html.SnarkHtml
-import space.kscience.snark.html.readSiteData
-import space.kscience.snark.ktor.site
-import kotlin.io.path.Path
-import kotlin.io.path.exists
-
-
-fun Application.snarkApplication(contextBuilder: ContextBuilder.() -> Unit = {}, site: HtmlSite) {
-
-    val context = Context {
-        plugin(SnarkHtml)
-        contextBuilder()
-    }
-
-    val snark = context.request(SnarkHtml)
-
-    val dataDirectoryString = environment.config.propertyOrNull("snark.dataDirectory")?.getString() ?: "data"
-
-    val dataDirectory = Path(dataDirectoryString)
-
-    if (!dataDirectory.exists()) {
-        error("Data directory at $dataDirectory is not resolved")
-    }
-
-    val siteData = snark.readSiteData(context) {
-        directory(snark.io, Name.EMPTY, dataDirectory)
-    }
-
-    siteData.forEach { namedData ->
-        log.debug("Loading data {} from {}", namedData.name, namedData.meta[FileData.FILE_PATH_KEY])
-    }
-
-    routing {
-        staticResources("/css","css")
-        site(context, siteData, content = site)
-    }
-}

gradle.properties

@@ -1,3 +1,3 @@
 kotlin.code.style=official
 
-toolsVersion=0.15.2-kotlin-1.9.22
+toolsVersion=0.15.4-kotlin-2.0.0

gradle/wrapper/gradle-wrapper.properties

@@ -1,5 +1,5 @@
 distributionBase=GRADLE_USER_HOME
 distributionPath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-8.6-bin.zip
+distributionUrl=https\://services.gradle.org/distributions/gradle-8.10.2-bin.zip
 zipStoreBase=GRADLE_USER_HOME
 zipStorePath=wrapper/dists

settings.gradle.kts

@@ -20,6 +20,10 @@ pluginManagement {
     }
 }
 
+plugins {
+    id("org.gradle.toolchains.foojay-resolver-convention") version "0.8.0"
+}
+
 dependencyResolutionManagement {
 
     val toolsVersion: String by extra

snark-core/README.md

@@ -0,0 +1,21 @@
+# Module snark-core
+
+
+
+## Usage
+
+## Artifact:
+
+The Maven coordinates of this project are `space.kscience:snark-core:0.2.0-dev-1`.
+
+**Gradle Kotlin DSL:**
+```kotlin
+repositories {
+    maven("https://repo.kotlin.link")
+    mavenCentral()
+}
+
+dependencies {
+    implementation("space.kscience:snark-core:0.2.0-dev-1")
+}
+```

snark-core/src/commonMain/kotlin/space/kscience/snark/Snark.kt

@@ -46,8 +46,8 @@ public class Snark : WorkspacePlugin() {
 
         override fun build(context: Context, meta: Meta): Snark = Snark()
 
-        private val byteArrayIOReader = IOReader {
-            readByteArray()
+        private val byteArrayIOReader: IOReader<ByteArray> = IOReader { source ->
+            source.readByteArray()
         }
 
         internal val byteArraySnarkParser = SnarkReader(byteArrayIOReader)

snark-gradle-plugin/README.md

@@ -0,0 +1,21 @@
+# Module snark-gradle-plugin
+
+
+
+## Usage
+
+## Artifact:
+
+The Maven coordinates of this project are `space.kscience:snark-gradle-plugin:0.2.0-dev-1`.
+
+**Gradle Kotlin DSL:**
+```kotlin
+repositories {
+    maven("https://repo.kotlin.link")
+    mavenCentral()
+}
+
+dependencies {
+    implementation("space.kscience:snark-gradle-plugin:0.2.0-dev-1")
+}
+```

snark-html/README.md

@@ -0,0 +1,32 @@
+# Module snark-html
+
+
+
+## Features
+
+ - [data](#) : Data-based processing. Instead of traditional layout-based
+ - [layouts](#) : Use custom layouts to represent a data tree
+ - [parsers](#) : Add custom file formats and parsers using DataForge dependency injection
+ - [preprocessor](#) : Preprocessing text files using templates
+ - [metadata](#) : Trademark DataForge metadata layering and transformations
+ - [dynamic](#) : Generating dynamic site using KTor server
+ - [static](#) : Generating static site
+
+
+## Usage
+
+## Artifact:
+
+The Maven coordinates of this project are `space.kscience:snark-html:0.2.0-dev-1`.
+
+**Gradle Kotlin DSL:**
+```kotlin
+repositories {
+    maven("https://repo.kotlin.link")
+    mavenCentral()
+}
+
+dependencies {
+    implementation("space.kscience:snark-html:0.2.0-dev-1")
+}
+```

snark-html/src/jvmMain/kotlin/space/kscience/snark/html/HtmlPage.kt

@@ -1,8 +1,11 @@
 package space.kscience.snark.html
 
+import kotlinx.css.html
 import kotlinx.html.HTML
-import kotlinx.html.stream.createHTML
-import kotlinx.html.visitTagAndFinalize
+import kotlinx.html.dom.append
+import kotlinx.html.dom.document
+import kotlinx.html.dom.serialize
+import kotlinx.html.html
 import space.kscience.dataforge.data.DataSink
 import space.kscience.dataforge.data.DataTree
 import space.kscience.dataforge.data.wrap
@@ -15,20 +18,22 @@ public fun interface HtmlPage {
     public fun renderPage()
 
     public companion object {
+
         public fun createHtmlString(
             pageContext: PageContext,
             dataSet: DataTree<*>?,
             page: HtmlPage,
-        ): String = createHTML().run {
-            HTML(kotlinx.html.emptyMap, this, null).visitTagAndFinalize(this) {
+        ): String = document {
+            append.html {
                 with(PageContextWithData(pageContext, dataSet ?: DataTree.EMPTY)) {
                     with(page) {
                         renderPage()
                     }
                 }
             }
-        }
+        }.serialize(true)
     }
+
 }
 
 

snark-html/src/jvmMain/kotlin/space/kscience/snark/html/MarkdownReader.kt

@@ -7,12 +7,11 @@ import org.intellij.markdown.MarkdownElementTypes
 import org.intellij.markdown.ast.ASTNode
 import org.intellij.markdown.ast.findChildOfType
 import org.intellij.markdown.ast.getTextInNode
+import org.intellij.markdown.flavours.commonmark.CommonMarkFlavourDescriptor
 import org.intellij.markdown.flavours.gfm.GFMFlavourDescriptor
-import org.intellij.markdown.flavours.space.SFMFlavourDescriptor
 import org.intellij.markdown.html.*
 import org.intellij.markdown.parser.LinkMap
 import org.intellij.markdown.parser.MarkdownParser
-import space.kscience.snark.SnarkReader
 
 private class SnarkInlineLinkGeneratingProvider(
     baseURI: URI?,
@@ -60,7 +59,7 @@ private class SnarkImageGeneratingProvider(
     }
 }
 
-public object SnarkFlavorDescriptor : GFMFlavourDescriptor(false) {
+public object SnarkFlavorDescriptor : CommonMarkFlavourDescriptor(false) {
     override fun createHtmlGeneratingProviders(linkMap: LinkMap, baseURI: URI?): Map<IElementType, GeneratingProvider> =
         super.createHtmlGeneratingProviders(linkMap, baseURI) + mapOf(
             MarkdownElementTypes.INLINE_LINK to SnarkInlineLinkGeneratingProvider(baseURI, absolutizeAnchorLinks)

snark-html/src/jvmMain/kotlin/space/kscience/snark/html/ParseAction.kt

@@ -1,68 +0,0 @@
-package space.kscience.snark.html
-
-import space.kscience.dataforge.actions.AbstractAction
-import space.kscience.dataforge.data.*
-import space.kscience.dataforge.io.Binary
-import space.kscience.dataforge.io.toByteArray
-import space.kscience.dataforge.meta.Meta
-import space.kscience.dataforge.meta.get
-import space.kscience.dataforge.misc.DFInternal
-import space.kscience.snark.SnarkReader
-import space.kscience.snark.TextProcessor
-import kotlin.coroutines.CoroutineContext
-import kotlin.coroutines.EmptyCoroutineContext
-import kotlin.reflect.KType
-import kotlin.reflect.typeOf
-
-@OptIn(DFInternal::class)
-internal fun <T, R> Data<T>.transform(
-    type: KType,
-    meta: Meta = this.meta,
-    coroutineContext: CoroutineContext = EmptyCoroutineContext,
-    block: suspend (T) -> R,
-): Data<R> {
-    val data = Data(type, meta, coroutineContext, listOf(this)) {
-        block(await())
-    }
-    return data
-}
-
-public class ParseAction(private val snarkHtml: SnarkHtml) :
-    AbstractAction<Binary, Any>(typeOf<PageFragment>()) {
-
-    private fun parseOne(data: NamedData<Binary>): NamedData<Any>? = with(snarkHtml) {
-        val contentType = getContentType(data.name, data.meta)
-
-        val parser: SnarkReader<Any>? = snark.readers.values.filter { parser ->
-            contentType in parser.inputContentTypes
-        }.maxByOrNull {
-            it.priority
-        }
-
-        //ignore data for which parser is not found
-        if (parser != null) {
-            val preprocessor = meta[TextProcessor.TEXT_PREPROCESSOR_KEY]?.let { snark.preprocessor(it) }
-            data.transform(parser.outputType) {
-                if (preprocessor == null) {
-                    parser.readFrom(it)
-                } else {
-                    //TODO provide encoding
-                    val string = it.toByteArray().decodeToString()
-                    parser.readFrom(preprocessor.process(string))
-                }
-            }.named(data.name)
-        } else {
-            null
-        }
-    }
-
-    override fun DataSink<Any>.generate(data: DataTree<Binary>, meta: Meta) {
-        data.forEach {
-            parseOne(it)?.let { put(it) }
-        }
-    }
-
-    override fun DataSink<Any>.update(source: DataTree<Binary>, meta: Meta, namedData: NamedData<Binary>) {
-        parseOne(namedData)?.let { put(it) }
-    }
-}

snark-html/src/jvmMain/kotlin/space/kscience/snark/html/SnarkHtml.kt

@@ -3,7 +3,6 @@
 package space.kscience.snark.html
 
 import io.ktor.http.ContentType
-import kotlinx.coroutines.CoroutineScope
 import kotlinx.io.readByteArray
 import space.kscience.dataforge.actions.Action
 import space.kscience.dataforge.actions.mapping
@@ -11,11 +10,11 @@ import space.kscience.dataforge.actions.transform
 import space.kscience.dataforge.context.Context
 import space.kscience.dataforge.context.PluginFactory
 import space.kscience.dataforge.context.PluginTag
-import space.kscience.dataforge.data.*
-import space.kscience.dataforge.io.Binary
-import space.kscience.dataforge.io.IOPlugin
-import space.kscience.dataforge.io.IOReader
-import space.kscience.dataforge.io.JsonMetaFormat
+import space.kscience.dataforge.data.DataSink
+import space.kscience.dataforge.data.DataTree
+import space.kscience.dataforge.data.filterByType
+import space.kscience.dataforge.data.putAll
+import space.kscience.dataforge.io.*
 import space.kscience.dataforge.io.yaml.YamlMetaFormat
 import space.kscience.dataforge.io.yaml.YamlPlugin
 import space.kscience.dataforge.meta.Meta
@@ -51,7 +50,7 @@ public class SnarkHtml : WorkspacePlugin() {
 
     override fun content(target: String): Map<Name, Any> = when (target) {
         SnarkReader::class.dfType -> mapOf(
-            "html".asName() to HtmlReader,
+            "html".asName() to RawHtmlReader,
             "markdown".asName() to MarkdownReader,
             "json".asName() to SnarkReader<Meta>(JsonMetaFormat, ContentType.Application.Json.toString()),
             "yaml".asName() to SnarkReader<Meta>(YamlMetaFormat, "text/yaml", "yaml"),
@@ -106,7 +105,34 @@ public class SnarkHtml : WorkspacePlugin() {
         }
     }
 
-    public val parseAction: Action<Binary, Any> = ParseAction(this)
+    public val parseAction: Action<Binary, Any> = Action.mapping {
+        val contentType = getContentType(name, meta)
+
+        val parser: SnarkReader<Any>? = snark.readers.values.filter { parser ->
+            contentType in parser.inputContentTypes
+        }.maxByOrNull {
+            it.priority
+        }
+
+        result(parser?.outputType ?: typeOf<Binary>()) { data ->
+
+            //ignore data for which parser is not found
+            if (parser != null) {
+                val preprocessor =
+                    meta[TextProcessor.TEXT_PREPROCESSOR_KEY]?.let { snark.preprocessor(it) }
+                if (preprocessor == null) {
+                    parser.readFrom(data)
+                } else {
+                    //TODO provide encoding
+                    val string = data.toByteArray().decodeToString()
+                    parser.readFrom(preprocessor.process(string))
+                }
+            } else {
+                data
+            }
+        }
+
+    }
 
     public val layoutAction: Action<Any, Any> = Action.mapping {
 
@@ -134,8 +160,8 @@ public class SnarkHtml : WorkspacePlugin() {
 
         override fun build(context: Context, meta: Meta): SnarkHtml = SnarkHtml()
 
-        private val byteArrayIOReader = IOReader {
-            readByteArray()
+        private val byteArrayIOReader = IOReader { source ->
+            source.readByteArray()
         }
 
         internal val byteArraySnarkParser = SnarkReader(byteArrayIOReader)
@@ -143,20 +169,24 @@ public class SnarkHtml : WorkspacePlugin() {
     }
 }
 
-
-public fun SnarkHtml.readSiteData(
+/**
+ * Parse raw data tree into html primitives
+ */
+public fun SnarkHtml.parseDataTree(
     binaries: DataTree<Binary>,
     meta: Meta = Meta.EMPTY,
-): DataTree<Any> = ObservableDataTree(context) {
+): DataTree<Any> = DataTree {
     //put all binaries
     putAll(binaries)
     //override ones which could be parsed
     putAll(binaries.transform(parseAction, meta))
 }.transform(prepareHeaderAction, meta).transform(removeIndexAction, meta)
 
-
-public fun SnarkHtml.readSiteData(
-    coroutineScope: CoroutineScope,
+/**
+ * Read the parsed data tree by providing [builder] for raw binary data tree
+ */
+public fun SnarkHtml.parseDataTree(
     meta: Meta = Meta.EMPTY,
+    //TODO add IO plugin as a context parameter
     builder: DataSink<Binary>.() -> Unit,
-): DataTree<Any> = readSiteData(ObservableDataTree(coroutineScope) { builder() }, meta)
+): DataTree<Any> = parseDataTree(DataTree { builder() }, meta)

snark-html/src/jvmMain/kotlin/space/kscience/snark/html/SnarkHtmlReader.kt

@@ -13,8 +13,8 @@ public interface SnarkHtmlReader : SnarkReader<PageFragment>{
     override val outputType: KType get() = typeOf<PageFragment>()
 }
 
-public object HtmlReader : SnarkHtmlReader {
-    override val inputContentTypes: Set<String> = setOf("html")
+public object RawHtmlReader : SnarkHtmlReader {
+    override val inputContentTypes: Set<String> = setOf("text/html", "html")
 
     override fun readFrom(source: String): PageFragment = PageFragment {
         div {

snark-html/src/jvmMain/resources/application.conf

@@ -1,12 +0,0 @@
-ktor {
-    application {
-        modules = [ ru.mipt.spc.ApplicationKt.spcModule ]
-    }
-
-    deployment {
-        port = 7080
-        watch = ["classes", "data/"]
-    }
-
-    development = true
-}

snark-html/src/jvmMain/resources/logback.xml

@@ -1,29 +0,0 @@
-<configuration>
-    <timestamp key="bySecond" datePattern="yyyyMMdd'T'HHmmss"/>
-
-    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
-        <encoder>
-            <pattern>%d{YYYY-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
-        </encoder>
-    </appender>
-
-    <root level="trace">
-        <appender-ref ref="STDOUT"/>
-    </root>
-
-    <appender name="FILE" class="ch.qos.logback.core.FileAppender">
-        <!-- use the previously created timestamp to create a uniquely
-             named log file -->
-        <file>logs/${bySecond}.txt</file>
-        <encoder>
-            <pattern>%d{YYYY-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
-        </encoder>
-    </appender>
-
-    <root level="DEBUG">
-        <appender-ref ref="FILE" />
-    </root>
-
-    <logger name="org.eclipse.jetty" level="INFO"/>
-    <logger name="io.netty" level="INFO"/>
-</configuration>

snark-ktor/README.md

@@ -0,0 +1,21 @@
+# Module snark-ktor
+
+
+
+## Usage
+
+## Artifact:
+
+The Maven coordinates of this project are `space.kscience:snark-ktor:0.2.0-dev-1`.
+
+**Gradle Kotlin DSL:**
+```kotlin
+repositories {
+    maven("https://repo.kotlin.link")
+    mavenCentral()
+}
+
+dependencies {
+    implementation("space.kscience:snark-ktor:0.2.0-dev-1")
+}
+```

snark-ktor/src/jvmMain/kotlin/space/kscience/snark/ktor/extractData.kt

@@ -18,6 +18,7 @@ private const val BUILD_DATE_FILE = "/buildDate"
  *
  * @return true if cache is valid and false if it is reset
  */
+@Deprecated("To be removed")
 fun Application.prepareSnarkDataCacheDirectory(dataPath: Path): Boolean {
 
     // Clear data directory if it is outdated

snark-ktor/src/jvmMain/kotlin/space/kscience/snark/ktor/snarkApplication.kt

@@ -0,0 +1,74 @@
+package space.kscience.snark.ktor
+
+import io.ktor.server.application.Application
+import io.ktor.server.application.log
+import io.ktor.server.http.content.staticResources
+import io.ktor.server.routing.Route
+import io.ktor.server.routing.application
+import io.ktor.server.routing.routing
+import space.kscience.dataforge.context.Context
+import space.kscience.dataforge.context.ContextBuilder
+import space.kscience.dataforge.context.request
+import space.kscience.dataforge.data.forEach
+import space.kscience.dataforge.names.Name
+import space.kscience.dataforge.workspace.FileData
+import space.kscience.dataforge.workspace.directory
+import space.kscience.snark.html.HtmlSite
+import space.kscience.snark.html.SnarkHtml
+import space.kscience.snark.html.parseDataTree
+import kotlin.io.path.Path
+import kotlin.io.path.exists
+
+private fun Application.defaultDataPath() = environment.config
+    .propertyOrNull("snark.dataDirectory")?.getString() ?: "data"
+
+/**
+ * Create a snark site at a given route. Uses [dataPath] as a path to data directory.
+ *
+ * The default [dataPath] is taken from "snark.dataDirectory" property.
+ * If not defined, use "data" directory in current work directory.
+ */
+public fun Route.site(
+    contextBuilder: ContextBuilder.() -> Unit = {},
+    dataPath: String = application.defaultDataPath(),
+    site: HtmlSite,
+) {
+
+    val context = Context {
+        plugin(SnarkHtml)
+        contextBuilder()
+    }
+
+    val snark = context.request(SnarkHtml)
+
+    val dataDirectory = Path(dataPath)
+
+    if (!dataDirectory.exists()) {
+        error("Data directory at $dataDirectory is not resolved")
+    }
+
+    val siteData = snark.parseDataTree {
+        directory(snark.io, Name.EMPTY, dataDirectory)
+    }
+
+    siteData.forEach { namedData ->
+        application.log.debug("Loading data {} from {}", namedData.name, namedData.meta[FileData.FILE_PATH_KEY])
+    }
+
+    staticResources("/css", "css")
+    site(context, siteData, content = site)
+
+}
+
+/**
+ * A Ktor module for snark application builder
+ */
+public fun Application.snarkApplication(
+    contextBuilder: ContextBuilder.() -> Unit = {},
+    dataPath: String = defaultDataPath(),
+    site: HtmlSite,
+) {
+    routing {
+        site(contextBuilder, dataPath, site)
+    }
+}

snark-pandoc/README.md

@@ -1,66 +1,4 @@
-## Examples
-### Simple converting
-Convert from INPUT_FILE to OUTPUT_FILE:
-```java
-PandocWrapper wrapper = new PandocWrapper();
-wrapper.use(p -> {
-        var command = new PandocCommandBuilder(List.of(INPUT_FILE), OUTPUT_FILE);
-        PandocWrapper.execute(command);
-        });
-```
-Equal to:
-```
-pandoc --output=OUTPUT_FILE INPUT_FILE
-```
-### Convert and set formats
-Convert from INPUT_FILE to OUTPUT_FILE and set INPUT_FORMAT and OUTPUT_FORMAT:
-```java
-PandocWrapper wrapper = new PandocWrapper();
-wrapper.use(p -> {
-        var command = new PandocCommandBuilder(List.of(INPUT_FILE), OUTPUT_FILE);
-        command.formatForm(INPUT_FORMAT);
-        command.formatTo(OUTPUT_FORMAT);
-        PandocWrapper.execute(command);
-        });
-```
-Equal to:
-```
-pandoc --output=OUTPUT_FILE --from=INPUT_FORMAT --to=OUTPUT_FORMAT INPUT_FILE
-```
-### Converting with options
-Convert from INPUT_FILE to standalone OUTPUT_FILE and set variable KEY to VALUE :
-```java
-PandocWrapper wrapper = new PandocWrapper();
-wrapper.use(p -> {
-        var command = new PandocCommandBuilder(List.of(INPUT_FILE), OUTPUT_FILE);
-        command.standalone();
-        command.setVariable(KEY, VALUE);
-        PandocWrapper.execute(command);
-        });
-```
-Equal to:
-```
-pandoc --output=OUTPUT_FILE --standalone --variable=KEY:VALUE INPUT_FILE
-```
+# Module snark-pandoc
+
 
-### Write output from pandoc to file
-Receive possible input formats in OUTPUT_FILE:
-```java
-PandocWrapper wrapper = new PandocWrapper();
-wrapper.use(p -> {
-        var command = new PandocCommandBuilder();
-        command.getInputFormats();
-        PandocWrapper.execute(command, OUTPUT_FILE);
-        });
-```
-Then in OUTPUT_FILE will be a list supported input formats, one per line.
 
-### Write errors from pandoc to file
-Receive all from error stream and exit code in ERROR_FILE and output in OUTPUT_FILE:
-```java
-PandocWrapper wrapper = new PandocWrapper();
-wrapper.use(p -> {
-        var command = new PandocCommandBuilder(List.of(INPUT_FILE), OUTPUT_FILE);
-        PandocWrapper.execute(command, OUTPUT_FILE, ERROR_FILE);
-        });
-```