Add document rendering

examples/document/build.gradle.kts

@@ -0,0 +1,33 @@
+plugins {
+    id("space.kscience.gradle.mpp")
+    application
+}
+
+application {
+    mainClass.set("Mainkt")
+
+    val isDevelopment: Boolean = project.ext.has("development")
+    applicationDefaultJvmArgs = listOf("-Dio.ktor.development=$isDevelopment", "-Xmx200M")
+}
+
+val snarkVersion: String by extra
+val ktorVersion = space.kscience.gradle.KScienceVersions.ktorVersion
+
+kscience {
+    jvm()
+    useContextReceivers()
+
+    jvmMain {
+        implementation(projects.snarkKtor)
+        implementation("io.ktor:ktor-server-cio:$ktorVersion")
+        implementation(spclibs.logback.classic)
+    }
+
+    jvmTest{
+        implementation("io.ktor:ktor-server-tests:$ktorVersion")
+    }
+}
+
+kotlin {
+    explicitApi = org.jetbrains.kotlin.gradle.dsl.ExplicitApiMode.Disabled
+}

examples/document/data/chapter1.md

@@ -0,0 +1,20 @@
+
+# Chapter ${section}
+
+Curabitur hendrerit hendrerit rutrum. Nullam elementum libero a nisi viverra aliquet. Sed ut urna a sem bibendum dictum. Cras non elit sit amet ex ultrices iaculis. Fusce lobortis lacinia fermentum. Fusce in metus id massa mollis consequat. Quisque non dolor quis orci gravida vulputate. Vivamus sed pellentesque orci. Sed aliquet malesuada rhoncus. Mauris id aliquet lorem.
+
+## Section ${section(2)}
+
+Maecenas at iaculis ipsum. Praesent maximus tristique magna eu faucibus. In tincidunt elementum pharetra. Nam scelerisque eros mattis, suscipit odio sit amet, efficitur mi. Etiam eleifend pulvinar erat a aliquet. Cras pellentesque tincidunt mi eget scelerisque. Proin eget ipsum a velit lobortis commodo. Nulla facilisi. Donec id pretium leo. Ut nec tortor sapien. Praesent vehicula dolor ut laoreet commodo. Pellentesque convallis, sapien et placerat luctus, tortor magna sodales sem, non tristique eros sem vel ipsum. Nulla vulputate accumsan nulla. Duis tempor, mi nec pharetra suscipit, sem odio sagittis mi, ut dignissim odio erat a dolor.
+
+## Section ${section(2)}
+
+In a quam nec turpis venenatis vehicula at ut lorem. Vestibulum tincidunt at velit laoreet sodales. Fusce fermentum enim sed lacinia fringilla. Nam et augue vitae felis sagittis consectetur in eget mauris. Fusce eget auctor turpis. Quisque at tristique nibh, id fringilla arcu. Cras sed finibus sapien.
+
+## Section ${section(2)}
+
+Praesent ullamcorper volutpat facilisis. Quisque posuere nisi sed nisl tempor, et sollicitudin justo imperdiet. Donec interdum auctor dui, quis dapibus lorem ullamcorper et. Sed aliquam, augue a tempus viverra, felis nulla convallis magna, a lacinia orci nisi id mauris. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Ut congue cursus quam, in vehicula nibh laoreet vel. Vivamus tincidunt sit amet dui quis mollis. Aliquam sed bibendum erat. Vivamus eget ante sed sapien volutpat luctus.
+
+## Section ${section(2)}
+
+Donec auctor quis libero eu cursus. Ut molestie varius massa, quis eleifend purus. Quisque elementum, magna id sollicitudin ullamcorper, arcu orci ullamcorper felis, ac tempus massa nisi vitae elit. In mollis porttitor orci. Integer gravida massa ut massa imperdiet, quis maximus libero varius. Phasellus vitae quam commodo, aliquet quam sed, euismod velit. Cras nibh ante, sodales non nulla sed, fringilla posuere orci.

examples/document/data/chapter2.md

@@ -0,0 +1,12 @@
+
+# Chapter ${section}
+
+Fusce at tristique ex. Proin vehicula venenatis mattis. Fusce at congue sapien, sed interdum lacus. Vivamus scelerisque ligula pretium nisl accumsan, molestie commodo sem condimentum. Nam ullamcorper leo quis sapien commodo, feugiat pellentesque purus rhoncus. Cras feugiat, lorem sit amet sodales aliquet, ante ipsum aliquam felis, ac rhoncus risus felis non enim. Suspendisse bibendum ornare efficitur. Nam tortor dolor, imperdiet nec orci et, pellentesque elementum sem. Integer sapien urna, rhoncus et felis et, fringilla euismod elit. Quisque tellus quam, tincidunt sed velit at, aliquam mollis leo. Integer pellentesque leo in libero pretium pharetra vitae sed ipsum. Mauris auctor venenatis pharetra. Maecenas tincidunt nulla ullamcorper, faucibus ante id, bibendum turpis. In lacus risus, pretium vel accumsan non, rhoncus nec augue. Suspendisse potenti.
+
+Orci varius natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Ut eu aliquet leo. Duis luctus viverra ex, at tincidunt diam fringilla at. Nunc rhoncus lorem arcu. Donec et nisi erat. Interdum et malesuada fames ac ante ipsum primis in faucibus. Sed vulputate lobortis velit. Etiam nisi sem, pellentesque sit amet molestie vel, porta vel enim. Cras sed diam sit amet nibh laoreet blandit non eget dui. Suspendisse tellus metus, pretium tempor euismod non, rhoncus eu enim. Etiam pharetra diam in quam auctor viverra. Nunc rhoncus libero quis dolor elementum accumsan. Mauris sed lectus fermentum, suscipit justo ac, faucibus tortor. Cras at volutpat enim, dapibus fringilla justo. Nulla vel erat quis neque congue laoreet.
+
+Integer et metus metus. Donec fringilla nec sem sit amet bibendum. Sed ornare lobortis velit eu gravida. Maecenas tincidunt ante et elit auctor convallis. Donec vestibulum augue et nisl fringilla aliquet venenatis ut diam. Nulla vitae leo est. Donec in magna blandit, dignissim ligula ultrices, cursus tortor. Praesent fermentum lorem placerat, venenatis ipsum eget, molestie ante. Duis non congue mi. Pellentesque non sem nibh. Donec feugiat lorem metus, sit amet dapibus augue congue vitae. Donec leo neque, sollicitudin et dignissim ac, semper vel mauris. Nunc fermentum egestas massa id varius. Aliquam interdum posuere mi in scelerisque. Aenean interdum consequat ultrices. Donec elementum tristique blandit.
+
+Cras finibus vel leo id mattis. Nulla tellus augue, bibendum in ipsum vitae, aliquet convallis nisl. Sed auctor urna sit amet ante pulvinar, sit amet venenatis nibh porta. Cras vitae ultrices nisi. Vestibulum eu sapien eu nulla rhoncus porttitor ut vitae odio. Curabitur scelerisque hendrerit elit vitae laoreet. Etiam eget accumsan nibh, non vehicula ex.
+
+Quisque ut ultricies nisi, eget vehicula ipsum. Quisque tortor mauris, sagittis vitae consectetur in, fermentum quis dui. Maecenas nec risus eu ipsum eleifend ornare. Nam tempus interdum mi, eget tincidunt enim interdum et. Nam a ultricies libero. Cras vehicula, quam a egestas semper, ipsum est cursus nunc, ac mollis est velit at enim. Sed nec nibh ut leo fermentum interdum. Donec aliquam elementum metus, non fermentum eros bibendum eu. Suspendisse ut odio vel dolor blandit condimentum. Vivamus malesuada accumsan magna, a suscipit tellus vehicula ut. Duis et orci arcu. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae;

examples/document/data/chapter3.md

@@ -0,0 +1,12 @@
+
+# Chapter ${section}
+
+Vestibulum mauris urna, sagittis in nibh placerat, vehicula suscipit leo. Maecenas lacinia varius tellus vel laoreet. Vestibulum sollicitudin nibh et nibh ullamcorper, at pretium orci molestie. Donec vulputate, nibh tempus maximus pretium, urna arcu consequat metus, vel ullamcorper est lectus id purus. Aliquam rutrum eu arcu nec convallis. Cras nec nisl ut massa tempus fringilla. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam non augue vel nisi commodo blandit a a augue. Sed aliquet semper ipsum nec maximus. Aliquam elementum tellus eu lectus tempus eleifend. Donec sapien nisi, maximus vitae ante nec, dictum vestibulum justo. Nam at ex est. Pellentesque nisi lacus, congue non felis posuere, pulvinar cursus justo. Fusce tincidunt dui vel pharetra fringilla.
+
+Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. In accumsan lobortis ante, vitae convallis eros lobortis vitae. Praesent ac mi id dui tempor pellentesque ac quis ligula. Ut tristique elit non ex euismod, ac hendrerit ante porta. Sed venenatis porttitor neque quis molestie. Aenean consectetur vehicula nisl quis tincidunt. Phasellus a eros tristique, rhoncus dui non, commodo ligula. Maecenas gravida, felis nec vulputate ornare, dolor mauris aliquet arcu, nec hendrerit lacus risus vitae arcu. Nullam dignissim nulla vulputate nisi pulvinar, non volutpat libero tristique. Nullam faucibus justo et elit condimentum sagittis. Ut imperdiet molestie purus.
+
+Fusce dignissim laoreet lectus ac sollicitudin. Cras porta dapibus orci vel vehicula. Integer dapibus vitae risus non luctus. Vestibulum eu orci nec tortor rutrum convallis tincidunt non lectus. Mauris aliquam, nunc id tincidunt cursus, turpis dui aliquam arcu, quis consectetur sapien nunc eget turpis. Donec leo metus, dictum vitae egestas ut, tristique ut sapien. Mauris tempus leo sit amet justo dignissim, a ullamcorper orci luctus.
+
+Nunc eget imperdiet lorem. Integer vitae vestibulum leo, vel tincidunt metus. Integer id lorem sodales, dignissim justo a, lacinia nisl. Maecenas vel nisi aliquet, vulputate tortor quis, pretium libero. Morbi convallis ex non vulputate scelerisque. Phasellus pharetra lacus in justo volutpat, vitae hendrerit leo porttitor. Suspendisse bibendum, dui eu ullamcorper mollis, magna arcu semper ipsum, sit amet ullamcorper lacus elit non ipsum. Nullam pulvinar justo a odio fringilla, sit amet pharetra odio consectetur. Integer tempus, lorem non tristique placerat, ipsum felis eleifend eros, fringilla feugiat velit felis egestas eros. Duis purus nibh, accumsan vitae purus eget, porttitor cursus quam. Aliquam eu eros in odio fringilla rhoncus nec sed nulla. Fusce ex ex, iaculis eu porttitor a, aliquam at ex. Pellentesque enim erat, egestas sit amet sodales quis, molestie ac ante. Phasellus fringilla tellus velit, eu elementum urna convallis ut. Nam condimentum gravida erat id efficitur.
+
+Nulla cursus aliquet justo malesuada dictum. Donec euismod rhoncus ex vel tempus. Praesent ac sodales massa, eu fringilla est. Phasellus ac sapien posuere, dapibus lacus eget, commodo sapien. Aliquam justo risus, posuere vitae erat non, volutpat eleifend dui. Morbi vel lobortis mauris. Duis non tortor vitae neque maximus porttitor. Aliquam ac placerat dolor. Duis id sollicitudin felis. Suspendisse ligula ex, convallis id velit quis, semper laoreet sem.

examples/document/src/jvmMain/kotlin/main.kt

@@ -0,0 +1,51 @@
+import io.ktor.server.application.Application
+import io.ktor.server.cio.CIO
+import io.ktor.server.engine.embeddedServer
+import io.ktor.server.routing.routing
+import space.kscience.dataforge.context.Context
+import space.kscience.dataforge.context.request
+import space.kscience.dataforge.names.Name
+import space.kscience.dataforge.names.asName
+import space.kscience.dataforge.workspace.directory
+import space.kscience.snark.html.SnarkHtml
+import space.kscience.snark.html.document.document
+import space.kscience.snark.html.document.fragment
+import space.kscience.snark.html.readSiteData
+import space.kscience.snark.ktor.site
+import kotlin.io.path.Path
+import kotlin.io.path.exists
+
+@Suppress("unused")
+fun Application.documents(context: Context) {
+    val snark = context.request(SnarkHtml)
+    val dataDirectory = Path("data")
+
+    if(!dataDirectory.exists()){
+        error("Data directory at $dataDirectory is not resolved")
+    }
+
+    val siteData = snark.readSiteData(context) {
+        directory(snark.io, Name.EMPTY, dataDirectory)
+    }
+
+    routing {
+        site(context, siteData){
+            document("loremIpsum".asName()){
+                fragment("chapter1")
+                fragment("chapter2")
+                fragment("chapter3")
+            }
+        }
+    }
+}
+
+
+fun main() {
+    val context = Context {
+        plugin(SnarkHtml)
+    }
+
+    embeddedServer(CIO) {
+        documents(context)
+    }.start(true)
+}

settings.gradle.kts

@@ -41,5 +41,6 @@ include(
     ":snark-core",
     ":snark-html",
     ":snark-ktor",
-    ":snark-pandoc"
+    ":snark-pandoc",
+    ":examples:document"
 )

snark-core/build.gradle.kts

@@ -8,8 +8,8 @@ val dataforgeVersion: String by rootProject.extra
 kscience{
     jvm()
     js()
+    useContextReceivers()
     dependencies{
         api("space.kscience:dataforge-workspace:$dataforgeVersion")
     }
-    useContextReceivers()
 }

snark-html/src/jvmMain/kotlin/space/kscience/snark/html/MarkdownReader.kt

@@ -84,7 +84,7 @@ public object MarkdownReader : SnarkHtmlReader {
 
     }
 
-    private val markdownFlavor = SnarkFlavorDescriptor//SFMFlavourDescriptor(false)
+    private val markdownFlavor = SnarkFlavorDescriptor
     private val markdownParser = MarkdownParser(markdownFlavor)
 
     override fun readFrom(source: Source): PageFragment = readFrom(source.readString())

snark-html/src/jvmMain/kotlin/space/kscience/snark/html/PageContext.kt

@@ -53,8 +53,6 @@ public interface PageContext : SnarkContext {
 
     /**
      * Resolve absolute url for a page with given [pageName].
-     *
-     * @param relative if true, add [SiteContext] route to the absolute page name
      */
     public fun resolvePageRef(pageName: Name, targetSite: SiteContext = site): String
 
@@ -66,6 +64,10 @@ context(PageContext)
 public val page: PageContext
     get() = this@PageContext
 
+context(PageContextWithData)
+public val page: PageContextWithData
+    get() = this@PageContextWithData
+
 public fun PageContext.resolvePageRef(pageName: String, targetSite: SiteContext = site): String =
     resolvePageRef(pageName.parseAsName(), targetSite)
 

snark-html/src/jvmMain/kotlin/space/kscience/snark/html/PageFragment.kt

@@ -71,8 +71,7 @@ context(SnarkContext)
 public fun DataTree<*>.resolveAllHtml(
     predicate: (name: Name, meta: Meta) -> Boolean,
 ): Map<Name, Data<PageFragment>> = filterByType<PageFragment> { name, meta, _ ->
-    predicate(name, meta)
+    predicate(name, meta) && meta["published"].string != "false"
-            && meta["published"].string != "false"
     //TODO add language confirmation
 }.asSequence().associate { it.name to it.data }
 

snark-html/src/jvmMain/kotlin/space/kscience/snark/html/Postprocessor.kt

@@ -36,15 +36,22 @@ public class WebPageTextProcessor(private val page: PageContext) : TextProcessor
                 page.pageMeta[nameString.parseAsName()].string ?: "@null"
             }
 
+            "siteMeta.get" -> {
+                val nameString = match.groups["name"]?.value
+                    ?: error("resolvePageRef requires a string (quoted) argument")
+                page.site.siteMeta[nameString.parseAsName()].string ?: "@null"
+            }
+
             else -> match.value
         }
-    }.replace(attributeRegex){ match->
+    }.replace(attributeRegex) { match ->
         val uri = URI(match.groups["uri"]!!.value)
-        val snarkUrl = when(uri.authority){
+        val snarkUrl = when (uri.authority) {
-            "homeRef"->page.homeRef
+            "homeRef" -> page.homeRef
             "ref" -> page.resolveRef(uri.path)
             "page" -> page.localisedPageRef(uri.path.parseAsName())
-            "meta" -> page.pageMeta[uri.path.parseAsName()].string ?: "@null"
+            "pageMeta" -> page.pageMeta[uri.path.parseAsName()].string ?: "@null"
+            "siteMeta" -> page.site.siteMeta[uri.path.parseAsName()].string ?: "@null"
             else -> match.value
         }
         "=\"$snarkUrl\""

snark-html/src/jvmMain/kotlin/space/kscience/snark/html/SiteContext.kt

@@ -105,6 +105,9 @@ context(SiteContext)
 public val site: SiteContext
     get() = this@SiteContext
 
+context(SiteContextWithData)
+public val site: SiteContextWithData
+    get() = this@SiteContextWithData
 
 /**
  * A wrapper for site context that allows convenient site building experience

snark-html/src/jvmMain/kotlin/space/kscience/snark/html/document/DocumentBuilder.kt

@@ -0,0 +1,81 @@
+package space.kscience.snark.html.document
+
+import kotlinx.coroutines.runBlocking
+import kotlinx.html.body
+import kotlinx.html.head
+import kotlinx.html.title
+import space.kscience.dataforge.data.*
+import space.kscience.dataforge.meta.Meta
+import space.kscience.dataforge.meta.get
+import space.kscience.dataforge.meta.string
+import space.kscience.dataforge.names.Name
+import space.kscience.snark.SnarkBuilder
+import space.kscience.snark.SnarkContext
+import space.kscience.snark.html.*
+import kotlin.reflect.typeOf
+
+/**
+ *  A context for building a single document
+ */
+@SnarkBuilder
+public interface DocumentBuilder : SnarkContext {
+
+    public val documentName: Name
+
+    public val documentMeta: Meta
+
+    public val data: DataTree<*>
+
+    public suspend fun fragment(fragment: Data<*>)
+
+    public suspend fun fragment(fragment: DocumentFragment, overrideMeta: Meta? = null)
+}
+
+context(SiteContextWithData)
+public suspend fun DocumentBuilder.fragment(fragmentName: Name) {
+    fragment(site.siteData[fragmentName] ?: error("Can't find data fragment for $fragmentName in site data."))
+}
+
+context(SiteContextWithData)
+public suspend fun DocumentBuilder.fragment(fragmentName: String) {
+    fragment(site.siteData[fragmentName] ?: error("Can't find data fragment for $fragmentName in site data."))
+}
+
+private class PageBasedDocumentBuilder(val page: PageContextWithData) : DocumentBuilder {
+    override val documentName: Name get() = page.pageRoute
+    override val documentMeta: Meta get() = page.pageMeta
+    override val data: DataTree<*> get() = page.data
+
+    val fragments = mutableListOf<PageFragment>()
+
+    override suspend fun fragment(fragment: DocumentFragment, overrideMeta: Meta?) {
+        TODO("Not yet implemented")
+    }
+
+    override suspend fun fragment(fragment: Data<*>) {
+        when (fragment.type) {
+            typeOf<PageFragment>() -> fragments.add(fragment.await() as PageFragment)
+            typeOf<DocumentFragment>() -> fragment(fragment.await() as DocumentFragment, data.meta)
+            typeOf<String>() -> fragment(TextDocumentFragment(fragment.await() as String, fragment.meta))
+            else -> error("Unsupported data type: ${fragment.type}")
+        }
+    }
+}
+
+public fun SiteContextWithData.document(
+    documentName: Name,
+    documentMeta: Meta = Meta.EMPTY,
+    block: suspend DocumentBuilder.() -> Unit,
+): Unit = page(documentName, documentMeta) {
+    val documentBuilder = runBlocking { PageBasedDocumentBuilder(page).apply { block() } }
+    head {
+        title(documentMeta["title"].string ?: "Snark document")
+    }
+    body {
+        postprocess(DocumentTextProcessor(documentBuilder)) {
+            documentBuilder.fragments.forEach {
+                fragment(it)
+            }
+        }
+    }
+}

snark-html/src/jvmMain/kotlin/space/kscience/snark/html/document/DocumentFragment.kt

@@ -0,0 +1,14 @@
+package space.kscience.snark.html.document
+
+import kotlinx.io.files.Path
+import space.kscience.dataforge.meta.Meta
+
+public sealed interface DocumentFragment{
+    public val meta: Meta
+}
+
+public class TextDocumentFragment(public val text: String, override val meta: Meta) : DocumentFragment
+
+public class ImageDocumentFragment(public val image: Path, override val meta: Meta) : DocumentFragment
+
+public class CompositeDocumentFragment(public val fragments: List<DocumentFragment>, override val meta: Meta) : DocumentFragment

snark-html/src/jvmMain/kotlin/space/kscience/snark/html/document/DocumentTextProcessor.kt

@@ -0,0 +1,94 @@
+package space.kscience.snark.html.document
+
+import space.kscience.dataforge.meta.int
+import space.kscience.dataforge.meta.string
+import space.kscience.dataforge.names.*
+import space.kscience.snark.TextProcessor
+import java.net.URI
+
+public class DocumentTextProcessor(public val document: DocumentBuilder) : TextProcessor {
+
+    private val counters = mutableMapOf<String, Int>()
+
+    private fun getCounter(counter: String): Int = counters[counter] ?: 1
+
+    private fun getAndIncrementCounter(counter: String): Int {
+        val currentCounter = counters[counter] ?: document.documentMeta[NameToken("counter", counter).asName()].int ?: 1
+        counters[counter] = currentCounter + 1
+        return currentCounter
+    }
+
+    private fun resetAndGet(counter: String, value: Int = 1): Int {
+        counters[counter] = value
+        return value
+    }
+
+    private fun ref(counter: String, value: Int): String {
+        //TODO replace by name parse in future
+        val token = Name.parse(counter).last()
+        return when (token.body) {
+            "section" -> {
+                val level = token.index?.toIntOrNull() ?: 1
+                (1..level).joinToString(separator = "_") {
+                    if (it == level) value.toString() else getCounter("section[$it]").toString()
+                }
+            }
+
+            else -> "snark_counter_${counter}_$value"
+        }
+    }
+
+    override fun process(text: CharSequence): String = text.replace(functionRegex) { match ->
+        when (match.groups["function"]?.value) {
+
+            "documentName" -> {
+                document.documentName.toStringUnescaped()
+            }
+
+            "label" -> {
+                val counter = match.groups["arg1"]?.value ?: "@default"
+                val value = getAndIncrementCounter(counter)
+                val ref = ref(counter, value)
+                //language=HTML
+                """<a class="snark-label" id="$ref" href="#$ref">$value</a>"""
+            }
+
+//            "ref" -> {
+//                val target = match.groups["arg1"]?.value
+//                when
+//            }
+
+            "section" -> {
+                val level: Int = match.groups["arg1"]?.value?.toIntOrNull() ?: 1
+                val counter = getAndIncrementCounter("section[$level]")
+                val ref = ref("section[$level]", counter)
+                //language=HTML
+                """<a class="snark-section snark-label" id="$ref" href = "#$ref">$counter</a>"""
+            }
+
+            "documentMeta.get" -> {
+                val nameString = match.groups["name"]?.value
+                    ?: error("resolvePageRef requires a string (quoted) argument")
+                document.documentMeta[nameString.parseAsName()].string ?: "@null"
+            }
+
+            else -> match.value
+        }
+    }.replace(attributeRegex) { match ->
+        val uri = URI(match.groups["uri"]!!.value)
+        val snarkUrl = when (uri.authority) {
+            "documentName" -> document.documentName.toStringUnescaped()
+//            "ref" -> page.resolveRef(uri.path)
+            "meta" -> document.documentMeta[uri.path.parseAsName()].string ?: "@null"
+            else -> match.value
+        }
+        "=\"$snarkUrl\""
+    }
+
+
+    public companion object {
+        internal val functionRegex =
+            """\$\{(?<function>\w*)(?:\((?<arg1>[^(),]*)(\s*,\s*(?<arg2>[^(),]*))?\))?\}""".toRegex()
+        private val attributeRegex = """="(?<uri>snark://([^"]*))"""".toRegex()
+    }
+}