◊(Local Yarn Code "Changes On Branch dc332affec65437f")

Login
About History Code Code Docs Road Map Issues
Hide Diff Patch Side-by-Side Diff

Changes In Branch doc-expansion Through [dc332aff] Excluding Merge-Ins

This is equivalent to a diff from 75502f96 to dc332aff

2020-02-19
07:25
Merge doc updates. Closes [2e658da] check-in: 2cd87113 user: joel tags: trunk
2020-02-14
22:46
Clean up HTML Snippets scribble doc check-in: e2a5ab96 user: joel tags: doc-expansion
22:25
Reorganize Pollen and cache stuff in scribble docs check-in: dc332aff user: joel tags: doc-expansion
2020-02-12
21:13
Use friendlier repo URL in code docs check-in: 63876c18 user: joel tags: doc-expansion
2020-02-10
21:15
Merge updates from trunk check-in: e52e53c8 user: joel tags: doc-expansion
17:28
Improved footer check-in: 75502f96 user: joel tags: trunk
2020-02-08
17:56
The footer is here. Closes [87f985fb5b] and [751a7ebc2a] check-in: 2c0b202b user: joel tags: trunk

Added code-docs/cache.scrbl version [aa0f66e5]. 

































































































































































































































1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

#lang scribble/manual

@; SPDX-License-Identifier: BlueOak-1.0.0
@; This file is licensed under the Blue Oak Model License 1.0.0.

@(require "scribble-helpers.rkt" scribble/example)

@(require (for-label deta
                     db
                     racket/base
                     racket/contract
                     sugar/coerce
                     pollen/template
                     "../dust.rkt"
                     "../crystalize.rkt"
                     "../cache.rkt"))

@(define example-eval (make-base-eval))
@(example-eval '(require "cache.rkt" txexpr))

@title[#:tag "cache-rkt"]{Cache}

@defmodule["cache.rkt" #:packages ()]

In this project there are several places – the blog, the footer on each page, the RSS feed, series
pages — where data from an amorphous group of Pollen documents is needed. This is what the cache is
for.

This module defines and provides the schema and database connection to the SQLite cache, and some
functions for retrieving records from the cache. Use these when you need quick access to pre-cooked
HTML.

@section{Cache database}

@defthing[cache-conn connection?]{
The database connection.
}

@defproc[(init-cache-db!) void?]{
Creates and initializes the SQLite database cache file (named @filepath{vitreous.sqlite} and located
in the project root folder) by running queries to create tables in the database if they do not
exist.

This function is called automatically in @seclink["pollen-rkt"]{@filepath{pollen.rkt}} whenever HTML
is the target output. }

@defparam[current-plain-title title-plain non-empty-string? #:value "void"]{
Contains (or sets) the “plain” title (i.e., with no HTML markup) for the current article based on
analysis done by @racket[parse-and-cache-article!]. If the article did not specify a title,
a default title is supplied. If the article contained a @racket[note] that used the
@code{#:disposition} attribute, the disposition-mark may be included in the title.

This is a weird parameter, and at some point I will probably get rid of it and have
@racket[parse-and-cache-article!] supply it as an extra return value instead.

@margin-note{Note that this needs to be called @emph{after} @racket[parse-and-cache-article!] in
order to get an up-to-date value.}
}

@section{Retrieving cached data}

Some of this looks a little wacky, but it’s a case of putting a little extra complextity into the
back end to make things simple on the front end. These functions are most commonly used inside the
@emph{body} of a Pollen document (i.e., series pages). 

@filebox["series/my-series.poly.pm"
@codeblock|{
#lang pollen

◊title{My New Series}

...some other content

◊(<listing-excerpt> articles+notes #:order 'asc)
}|
]

@deftogether[(@defproc[(<listing-full> 
                                 [query-func (-> any/c query?)]
                                 [#:series series (or/c string? (listof string?) boolean?) #t]
                                 [#:limit limit integer? -1]
                                 [order stringish? 'desc]) txexpr?]
              @defproc[(<listing-excerpt>
                                 [query-func (-> any/c query?)]
                                 [#:series series (or/c string? (listof string?) boolean?) #t]
                                 [#:limit limit integer? -1]
                                 [order stringish? 'desc]) txexpr?]
              @defproc[(<listing-short>
                                 [query-func (-> any/c query?)]
                                 [#:series series (or/c string? (listof string?) boolean?) #t]
                                 [#:limit limit integer? -1]
                                 [order stringish? 'desc]) txexpr?])]{
Fetches the HTML for items from the SQLite cache, concatenates their HTML strings and returns
a @racket['style] tagged X-expression with this string as its element. The items will be ordered by
publish date according to @racket[_order] and optionally limited to the series specified in
@racket[_series].

The @racket[_query-func] should be either @racket[articles], which will create a listing of articles
only, or @racket[articles+notes], which will include notes intermingled with articles.

@margin-note{Note that the signature shown for the @racket[_query-func] argument above is
incomplete. If you choose to pass a function other than @racket[articles] or
@racket[articles+notes], you must use a function with exactly the same signature as those
functions.}

If @racket[_series] expression evaluates to @racket[#f], articles will not be filtered by series. If
it evaluates to @racket[#t] (the default), articles will be filtered by those that specify the
current output of @racket[here-output-path] in their @tt{series_pagenode} column in the SQLite
cache. If a string is supplied, articles will be filtered by those containing that exact value in
their @tt{series_pagenode} column in the SQLite cache.

The @racket[_order] expression must evaluate to either @racket["ASC"] or @racket["DESC"] and the
@racket[_limit] expressions must evaluate to a value suitable for use in the @tt{LIMIT} clause of
@ext-link["https://sqlite.org/lang_select.html"]{a SQLite @tt{SELECT} statement}. An expression that
evaluates to a negative integer (the default) is the same as having no limit.

The reason for enclosing the results in a @racket['style] txexpr is to prevent the HTML from being
escaped by @racket[->html] in the template. This tag was picked for the job because there will
generally never be a need to include any actual CSS information inside a @tt{<style>} tag in any
page, so it can be safely filtered out later. To remove the enclosing @tt{<style>} tag, see
@racket[unfence].
}

@defproc[(listing-htmls [listing-query query?]) (listof string?)]{
Returns the HTML bodies for the articles and/or notes returned by @racket[_listing-query] as a list
of strings.
}

@deftogether[(@defproc[(articles [type (or/c 'full 'excerpt 'short)]
                                 [#:series series (or/c string? (listof string?) boolean?) #t]
                                 [#:limit limit integer? -1]
                                 [order stringish? 'desc]) query?]
              @defproc[(articles+notes [type (or/c 'full 'excerpt 'short)]
                                       [#:series series (or/c string? (listof string?) boolean?) #t]
                                       [#:limit limit integer? -1]
                                       [order stringish? 'desc]) query?])]{
Create a query that fetches either articles only (@racket[articles]) or articles and notes
intermingled (@racket[articles+notes]), sorted by publish date and optionally limited to
a particular series.

Typically you will pass these functions by name to listing functions like @racket[<listing-full>]
rather than calling them directly.

@examples[#:eval example-eval
(articles 'full)]
}

@defproc[(unfence [html string?]) string?]{
Returns @racket[_html] with all occurrences of @racket["<style>"] and @racket["</style>"] removed.
The contents of the style tags are left intact.

Use this in templates with strings returned from @racket[->html] when called on docs that use the
@racket[<listing-full>] tag function or its siblings. }

@defproc[(series-grouped-list) (listof (listof cache:series?))]{
Return a list of lists of all @racket[cache:series] in the cache database. The series are grouped so
that series using the same value in the @tt{noun-plural} column appear together.}

@section{Deleting records}

@deftogether[(@defproc[(delete-article! [page stringish?]) void?]
              @defproc[(delete-notes!   [page stringish?]) void?])]{
Delete a particular article, or all notes for a particular article, respectively.
}

@section{Schema}

The cache database has four tables: @tt{articles}, @tt{notes}, @tt{index_entries} and @tt{series}. Each of these has a corresponding schema, shown below. In addition, there is a “virtual” schema, @tt{listing}, for use with queries which may or may not combine articles and notes intermingled.

The work of picking apart an article’s exported @tt{doc} and @tt{metas} into rows in these tables is done by @racket[parse-and-cache-article!].

The below are shown as @code{struct} forms but are actually defined with deta’s @racket[define-schema]. Each schema has an associated struct with the same name and a smart constructor called @tt{make-@emph{id}}. The struct’s “dumb” constructor is hidden so that invalid entities cannot be created. For every defined field there is an associated functional setter and updater named @tt{set-@emph{id}-field} and @tt{update-@emph{id}-field}, respectively.

@defstruct*[cache:article ([id                   id/f]
                           [page                 symbol/f]
                           [title-plain          string/f]
                           [title-html-flow      string/f]
                           [title-specified      boolean/f]
                           [published            string/f]
                           [updated              string/f]
                           [author               string/f]
                           [conceal              string/f]
                           [series-page          string/f]
                           [noun-singular        string/f]
                           [note-count           integer/f]
                           [doc-html             string/f]
                           [disposition          string/f]
                           [disp-html-anchor     string/f]
                           [listing-full-html    string/f]
                           [listing-excerpt-html string/f]
                           [listing-short-html   string/f])
            #:constructor-name make-cache:article]{
Table holding cached article information.
}

@defstruct*[cache:note ([id              id/f]
                        [page            symbol/f]
                        [html-anchor     string/f]
                        [title-html-flow string/f]
                        [title-plain     string/f]
                        [author          string/f]
                        [author-url      string/f]
                        [published       string/f]
                        [disposition     string/f]
                        [content-html    string/f]
                        [series-page     symbol/f]
                        [conceal         string/f]
                        [listing-full-html    string/f]
                        [listing-excerpt-html string/f]
                        [listing-short-html   string/f])
                       #:constructor-name make-cache:note]{
Table holding cached information on notes.
}

@defstruct*[cache:series ([id            id/f]
                          [page          symbol/f]
                          [title         string/f]
                          [published     string/f]
                          [noun-plural   string/f]
                          [noun-singular string/f])
                         #:constructor-name make-cache:series]{
Table holding cached-information on series.
}








Modified code-docs/crystalize.scrbl
from [146fa5fb]
to [af1f96a5].



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127

1
2
3
4
5
6
7
8
9
10
11
12
13

14


15

16
17
18
19
20
21
22
23
24
25
26








27
28
29
30
31
32
33
34
35
36
37
38




























































































-

-
-
+
-











-
-
-
-
-
-
-
-












-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-

#lang scribble/manual

@; SPDX-License-Identifier: BlueOak-1.0.0
@; This file is licensed under the Blue Oak Model License 1.0.0.

@(require "scribble-helpers.rkt")

@(require (for-label "../pollen.rkt"
                     "../dust.rkt"
                     "../crystalize.rkt"
                     racket/base
                     racket/contract
                     racket/string
                     deta
                     txexpr
                     pollen/template
                     pollen/pagetree
                     pollen/pagetree))
                     sugar/coerce))

@title{@filepath{crystalize.rkt}}

@defmodule["crystalize.rkt" #:packages ()]

“Crystalizing” is an extra layer in between docs and templates that destructures the doc and stores
it in various pieces in a SQLite cache. Individual articles save chunks of rendered HTML to the
cache when their individual pages are rendered. When pulling together listings of articles in
different contexts that need to be filtered and sorted, a SQL query is much faster than trolling
through the Pollen cache for matching docs and regenerating the HTML.

@defproc[(init-cache-db!) void?]

Initializes the SQLite database cache file (named @filepath{vitreous.sqlite} and located in the
project root folder) by running queries to create tables in the database if they do not exist. (The
file itself is created at the module level.)

This function is called automatically in @filepath{pollen.rkt} whenever HTML is the target output.

@defproc[(parse-and-cache-article! [pagenode pagenode?] [doc txexpr?]) non-empty-string?]

Returns a string containing the HTML of @racket[_doc]. @margin-note{This is one function that breaks
my convention of using a prefix of @tt{html$-} for functions that return a single string of HTML.}

Privately, it does a lot of other work. The article is analyzed, additional metadata is constructed,
and it is saved to the SQLite cache. If the article specifies a @racket['series] meta, information
about that series is fetched and used in the rendering of the article. If there are @racket[note]s
in the doc, they are parsed and saved individually to the SQLite cache. If any of the notes use the
@code{#:disposition} attribute, information about the disposition is parsed out and used in the
rendering of the article.

@deftogether[(@defproc[(<listing-full> 
                                 [query-func (-> any/c query?)]
                                 [#:series series (or/c string? (listof string? boolean?)) #t]
                                 [#:limit limit integer? -1]
                                 [order stringish? 'desc]) txexpr?]
              @defproc[(<listing-excerpt>
                                 [query-func (-> any/c query?)]
                                 [#:series series (or/c string? (listof string? boolean?)) #t]
                                 [#:limit limit integer? -1]
                                 [order stringish? 'desc]) txexpr?]
              @defproc[(<listing-short>
                                 [query-func (-> any/c query?)]
                                 [#:series series (or/c string? (listof string? boolean?)) #t]
                                 [#:limit limit integer? -1]
                                 [order stringish? 'desc]) txexpr?])]

Fetches the HTML for items from the SQLite cache and returns the HTML strings fenced inside
a @racket['style] tagged X-expression. The items will be ordered by publish date according to
@racket[_order] and optionally limited to the series specified in @racket[_series].

The @racket[_query-func] should be either @racket[articles], which will create a listing of articles
only, or @racket[articles+notes], which will include notes intermingled with articles.

@margin-note{Note that the signature shown for the @racket[_query-func] argument above is
incomplete. If you choose to pass a function other than @racket[articles] or
@racket[articles+notes], you must use a function with exactly the same signature as those
functions.}

If @racket[_series] expression evaluates to @racket[#f], articles will not be filtered by series. If
it evaluates to @racket[#t] (the default), articles will be filtered by those that specify the
current output of @racket[here-output-path] in their @tt{series_pagenode} column in the SQLite
cache. If a string is supplied, articles will be filtered by those containing that exact value in
their @tt{series_pagenode} column in the SQLite cache.

The @racket[_order] expression must evaluate to either @racket["ASC"] or @racket["DESC"] and the
@racket[_limit] expressions must evaluate to a value suitable for use in the @tt{LIMIT} clause of
@ext-link["https://sqlite.org/lang_select.html"]{a SQLite @tt{SELECT} statement}. An expression that
evaluates to a negative integer (the default) is the same as having no limit.

The reason for enclosing the results in a @racket['style] txexpr is to prevent the HTML from being
escaped by @racket[->html] in the template. This tag was picked for the job because there will
generally never be a need to include any actual CSS information inside a @tt{<style>} tag in any
page, so it can be safely filtered out later. To remove the enclosing @tt{<style>} tag, see
@racket[unfence].

@deftogether[(@defproc[(articles [type (or/c 'full 'excerpt 'short)]
                                 [#:series series (or/c string? (listof string? boolean?)) #t]
                                 [#:limit limit integer? -1]
                                 [order stringish? 'desc]) query?]
              @defproc[(articles+notes [type (or/c 'full 'excerpt 'short)]
                                       [#:series series (or/c string? (listof string? boolean?)) #t]
                                       [#:limit limit integer? -1]
                                       [order stringish? 'desc]) query?])]

Create a query that fetches either articles only (@racket[articles]) or articles and notes
intermingled (@racket[articles+notes]), sorted by publish date and optionally limited to
a particular series.

Typically you will pass these functions by name to listing functions like @racket[<listing-full>]
rather than calling them directly.

@defproc[(unfence [html string?]) string?]

Returns @racket[_html] with all occurrences of @racket["<style>"] and @racket["</style>"] removed.
The contents of the style tags are left intact.

Use this with strings returned from @racket[->html] when called on docs that use the
@racket[<listing-full>] tag function or its siblings.

@defparam[current-plain-title non-empty-string? #:value "void"]

Contains (or sets) the “plain” title (i.e., with no HTML markup) for the current article based on
analysis done by @racket[parse-and-cache-article!]. If the article did not specify a title,
a default title is supplied. If the article contained a @racket[note] that used the
@code{#:disposition} attribute, the disposition-mark may be included in the title.

Note that this needs to be called @emph{after} @racket[parse-and-cache-article!] in order to get an
up-to-date value.








Added code-docs/custom.css version [471eaf45].
































1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

.fileblock .SCodeFlow {
    padding-top: 0.7em;
    margin-top: 0;
}

.fileblock {
    width: 90%;
}

.fileblock_filetitle{
    background: #eee;
    text-align:right;
    padding: 0.15em;
    border: 1px dotted black;
    border-bottom: none;
}

.terminal, .browser {
    margin-bottom: 1em;
    padding: 0.5em;
    width: 88%;
    background: #fcfcfc;
    color: rgb(150, 35, 105);
}

.terminal .SIntrapara, .browser .SIntrapara, .fileblock .SIntrapara {
    margin: 0 0 0 0;
}








Added code-docs/design.scrbl version [7f7e48a9].













































1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

#lang scribble/manual

@; SPDX-License-Identifier: BlueOak-1.0.0
@; This file is licensed under the Blue Oak Model License 1.0.0.

@(require "scribble-helpers.rkt"
          racket/runtime-path)

@(require (for-label racket/base))

@title{Design Background}

The design and implementation of @italic{The Local Yarn} are guided by requirements that have
evolved since I started the site in 1999. I enumerate them here because they explain why the code is
necessarily more complicated than a typical blog.

@itemlist[
  @item{@bold{The writing will publish to two places from the same source: the web server, and the
  bookshelf.} The web server, because it’s a fun, fast way to publish writing and code to the whole
  world (you knew that already); but also on bookshelves, because
  @ext-link["https://thelocalyarn.com/excursus/secretary/posts/web-books.html"]{a web server is like
  a projector}, and I want to be able to turn it off someday and still have something to show for
  all my work. Plus, I just like printed books.}

  @item{@bold{Changes are part of the content.} I like to revisit, resurface and amend things I’ve
  written before. Views change, new ideas come along. In a typical blog the focus is always at
  whatever’s happening at the head of the time stream; an addendum to an older post is, for all
  practical purposes, invisible and nearly useless. I want every published edit to an article to be
  findable and linkable. I want addenda to be extremely visible. These addenda should also be able
  to mark major shifts in the author’s own perspective on what they originally wrote.}

  @item{@bold{Everything produced here, both in print and on the web, should look good.}
  @item{@bold{All the output should be produced (and reproducible) by automated processes.} No
  clicking on buttons in apps to publish web pages or books.}
]

@nested[#:style 'inset]{
  “Yet modest ornament with use combined @(linebreak)
  Attracts the eye to exercise the mind.” @(linebreak)
  —@ext-link["https://en.wikipedia.org/wiki/Samuel_Rogers"]{Samuel Rogers}
}








Modified code-docs/dust.scrbl
from [dfcfb9ab]
to [c0b36b73].



1
2
3
4
5
6
7
8
9
10

11
12
13
14
15
16
17
18
19
20
21
22
23

24
25
26
27
28
29
30

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

24
25
26
27
28
29
30
31











+












-
+








#lang scribble/manual

@; SPDX-License-Identifier: BlueOak-1.0.0
@; This file is licensed under the Blue Oak Model License 1.0.0.

@(require "scribble-helpers.rkt"
          scribble/example)

@(require (for-label "../pollen.rkt"
                     "../dust.rkt"
                     "../cache.rkt"
                     racket/base
                     racket/contract
                     txexpr
                     sugar/coerce
                     pollen/tag
                     pollen/setup
                     pollen/pagetree
                     pollen/core))

@(define dust-eval (make-base-eval))
@(dust-eval '(require "dust.rkt" txexpr))

@title{@filepath{dust.rkt}}
@title{Dust}

@defmodule["dust.rkt" #:packages ()]

This is where I put constants and helper functions that are needed pretty much everywhere in the
project. In a simpler project these would go in @seclink["pollen-rkt"]{@filepath{pollen.rkt}} but
here I have other modules sitting “behind” that one in the @tt{require} chain.





︙


140
141
142
143
144
145
146








147
148
149
150
151
152
153
154


155
156
157
158
159
160
161

141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161


162
163
164
165
166
167
168
169
170








+
+
+
+
+
+
+
+






-
-
+
+








(first-words txs-punc-and-split-elems 5)
(first-words txs-dashes 5)
(first-words txs-parens-commas 5)
(first-words txs-short 5)
]

@section{Article parsers and helpers}

@defparam[listing-context ctxt (or/c 'blog 'feed 'print "") #:value ""]

A parameter specifying the current context where any listings of articles would appear. Its purpose
is to allow articles to exclude themselves from certain special collections (e.g., the blog, the RSS
feed, print editions). Any article whose @code{conceal} meta matches the current context will not be
included in any listings returned by the listing functions in
@seclink["cache-rkt"]{@filepath{cache.rkt}}.

@defproc[(default-title [body-txprs (listof txexpr?)]) string?]

Given a list of tagged X-expressions (the elements of an article’s doc, e.g.), returns a string
containing a suitable title for the document. (Uses @racket[first-words].)

Titles are not required for articles, but there are contexts where you need something that
serves as a title if one is not present, and that’s what this function supplies.
Titles are not required for articles, but there are contexts where you need something that serves as
a title if one is not present, and that’s what this function supplies.

@examples[#:eval dust-eval
(define doc 
  '(root (p "If I had been astonished at first catching a glimpse of so outlandish an "
            "individual as Queequeg circulating among the polite society of a civilized "
            "town, that astonishment soon departed upon taking my first daylight "
            "stroll through the streets of New Bedford…")))




︙







Modified code-docs/main.scrbl
from [6cbd40b5]
to [33fc9294].




︙


9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25








26

27
28
29

30
31
32
33
34
35



9
10
11
12
13
14
15

16
17
18






19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


37
38








-



-
-
-
-
-
-
+
+
+
+
+
+
+
+

+



+




-
-
+
+


@author{Joel Dueck}

These are my notes about the internals of the Local Yarn source code. In other words, a personal
reference, rather than a tutorial. These pages concern only the source code itself. Refer to the
wiki for info about deployment, etc.


You’ll get the most out of these notes if you have read @other-doc['(lib
"pollen/scribblings/pollen.scrbl")], and worked through the tutorials by hand.

If you’re viewing these notes on the Fossil repository, note that these pages are heavily
interlinked with the central Racket documentation at @tt{docs.racket-lang.org}, which are written
and maintained by others. Links on those pages that lead outside of that domain will not work within
this repo’s “Code Docs” frame, due to the repository’s
@ext-link["https://content-security-policy.com"]{content security policy}. To follow such links,
right-click and open the link in a new tab or window.
@margin-note{Note that these pages are heavily interlinked with the central Racket documentation at
@tt{docs.racket-lang.org}, which are written and maintained by others. 

If you’re browsing these docs from within @italic{The Local Yarn}’s main website, and if you follow
links to other Racket documentation, you’ll find that to @emph{other} sites on those pages will not
work (due to the  @ext-link["https://content-security-policy.com"]{content security policy} in
effect when inside a frame). To follow such links, right-click and open the link in a new tab or
window.

You may also wish to @ext-link["#"]{open this page in its own tab.}}

@local-table-of-contents[]

@include-section["tour.scrbl"]
@include-section["overview.scrbl"]
@include-section["pollen.scrbl"]  @; pollen.rkt
@include-section["dust.scrbl"]    @; dust.rkt
@include-section["snippets-html.scrbl"] @; you get the idea
@include-section["crystalize.scrbl"]

@include-section["cache.scrbl"]
@include-section["crystalize.scrbl"]








Modified code-docs/pollen.scrbl
from [c9accf05]
to [ecc33834].




︙


12
13
14
15
16
17
18
19

20
21
22
23
24
25

26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78

12
13
14
15
16
17
18

19
20
21
22
23
24

25
26
27
28
29










































30
31
32
33
34
35
36








-
+





-
+




-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-








                     racket/string
                     txexpr
                     pollen/tag
                     pollen/setup
                     pollen/core
                     sugar/coerce))

@title[#:tag "pollen-rkt"]{@filepath{pollen.rkt}}
@title[#:tag "pollen-rkt"]{Pollen}

@defmodule["pollen.rkt" #:packages ()]

The file @filepath{pollen.rkt} is implicitly @code{require}d in every template and every @code{#lang
pollen} file in the project. It defines the markup for all Pollen documents, and also re-provides
everything provided by @code{crystalize.rkt}.
everything provided by @filepath{cache.rkt} and @filepath{crystalize.rkt}.

The @code{setup} module towards the top of the file is used as described in
@racketmodname[pollen/setup].

@section{Defining new tags}

I use a couple of macros to define tag functions that automatically branch into other functions
depending on the current output target format. This allows me to put the format-specific tag
functions in separate files that have separate places in the dependency chain. So if only the HTML
tag functions have changed and not those for PDF, the makefile can ensure only the HTML files are
rebuilt.

@defproc[#:kind "syntax"
 (poly-branch-tag (tag-id symbol?))
 (-> txexpr?)]

Defines a new function @racket[_tag-id] which will automatically pass all of its arguments to a
function whose name is the value returned by @racket[current-poly-target], followed by a hyphen,
followed by @racket[_tag]. So whenever the current output format is @racket['html], the function
defined by @racket[(poly-branch-tag _p)] will branch to a function named @racket[html-p]; when the
current format is @racket['pdf], it will branch to @racket[pdf-p], and so forth.

You @emph{must} define these branch functions separately, and you must define one for @emph{every}
output format included in the definition of @racket[poly-targets] in this file’s @racket[setup]
submodule. If you do not, you will get “unbound identifier” errors at expansion time.

The convention in this project is to define and provide these branch functions in separate files:
see, e.g., @filepath{tags-html.rkt}.

Functions defined with this macro @emph{do not} accept keyword arguments. If you need keyword
arguments, see @racket[poly-branch-kwargs-tag].

@margin-note{The thought behind having two macros so similar is that, by cutting out handling for keyword
arguments, @racket[poly-branch-tag] could produce simpler and faster code. I have not verified if
this intuition is meaningful or correct.}

@defproc[#:kind "syntax"
         (poly-branch-kwargs-tag (tag-id symbol?))
         (-> txexpr?)]

Works just like @racket[poly-branch-tag], but uses Pollen’s @racket[define-tag-function] so that
keyword arguments will automatically be parsed as X-expression attributes.

Additionally, the branch functions called from the new function must accept exactly two arguments:
a list of attributes and a list of elements.

@section{Markup reference}

These are the tags that can be used in any of @italic{The Local Yarn}’s Pollen documents (articles,
etc).

@defproc[(title [element xexpr?] ...) txexpr?]





︙


293
294
295
296
297
298
299











































251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299








+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

#lang pollen

◊for/s[x '(7 8 9)]{Now once for number ◊x}

◊;Above line is shorthand for this one:
◊for/splice[[(x (in-list '(7 8 9)))]]{Now once for number ◊x}
}|

@section{Defining new tags}

I use a couple of macros to define tag functions that automatically branch into other functions
depending on the current output target format. This allows me to put the format-specific tag
functions in separate files that have separate places in the dependency chain. So if only the HTML
tag functions have changed and not those for PDF, the @filepath{makefile} can ensure only the HTML files are
rebuilt.

@defproc[#:kind "syntax"
 (poly-branch-tag (tag-id symbol?))
 (-> txexpr?)]

Defines a new function @racket[_tag-id] which will automatically pass all of its arguments to a
function whose name is the value returned by @racket[current-poly-target], followed by a hyphen,
followed by @racket[_tag]. So whenever the current output format is @racket['html], the function
defined by @racket[(poly-branch-tag _p)] will branch to a function named @racket[html-p]; when the
current format is @racket['pdf], it will branch to @racket[pdf-p], and so forth.

You @emph{must} define these branch functions separately, and you must define one for @emph{every}
output format included in the definition of @racket[poly-targets] in this file’s @racket[setup]
submodule. If you do not, you will get “unbound identifier” errors at expansion time.

The convention in this project is to define and provide these branch functions in separate files:
see, e.g., @filepath{tags-html.rkt}.

Functions defined with this macro @emph{do not} accept keyword arguments. If you need keyword
arguments, see @racket[poly-branch-kwargs-tag].

@margin-note{The thought behind having two macros so similar is that, by cutting out handling for keyword
arguments, @racket[poly-branch-tag] could produce simpler and faster code. I have not verified if
this intuition is meaningful or correct.}

@defproc[#:kind "syntax"
         (poly-branch-kwargs-tag (tag-id symbol?))
         (-> txexpr?)]

Works just like @racket[poly-branch-tag], but uses Pollen’s @racket[define-tag-function] so that
keyword arguments will automatically be parsed as X-expression attributes.

Additionally, the branch functions called from the new function must accept exactly two arguments:
a list of attributes and a list of elements.








Modified code-docs/scribble-helpers.rkt
from [95058fad]
to [a1dd7f15].



1
2
3
4
5
6
7
8
9
10



11
12
13


14

15
16
17
18
19
20
21

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

19
20
21
22
23
24
25
26











+
+
+



+
+
-
+








#lang racket/base

; SPDX-License-Identifier: BlueOak-1.0.0
; This file is licensed under the Blue Oak Model License 1.0.0.

;; Convenience/helper functions for this project’s Scribble documentation

(require scribble/core
         scribble/manual/lang
         scribble/html-properties
         scribble/private/manual-sprop
         scribble/decode
         racket/runtime-path
         (only-in net/uri-codec uri-encode))
(provide (all-defined-out))

(define-runtime-path custom-css "custom.css")

(define repo-url/ "https://thelocalyarn.com/cgi-bin/yarncode/")
(define repo-url/ "https://thelocalyarn.com/code/")

;; Link to a ticket on the Fossil repository by specifying the ticket ID.
;; The "_parent" target breaks out of the iframe used by the Fossil repo web UI.
(define (ticket id-str)
  (hyperlink (string-append repo-url/ "tktview?name=" id-str)
             "ticket "
             (tt id-str)




︙


41
42
43
44
45
46
47




























46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79








+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

             #:style (style #f (list (attributes '((target . "_parent")))))))

(define (responsive-retina-image img-path)
  (image img-path 
         #:scale 0.5
         #:style (style #f (list (attributes '((style . "max-width:100%;height:auto;")))))))

;;
;; From https://github.com/mbutterick/pollen/blob/master/pollen/scribblings/mb-tools.rkt
;;

(define (terminal . args)
  (compound-paragraph (style "terminal" (list (css-style-addition custom-css) (alt-tag "div")))
                      (list (apply verbatim args))))

(define (cmd . args)
  (elem #:style (style #f (list (color-property "black"))) (tt args)))

(define (fileblock filename . inside)
  (compound-paragraph 
   (style "fileblock" (list* (alt-tag "div") 'multicommand
                             (box-mode "RfileboxBoxT" "RfileboxBoxC" "RfileboxBoxB") 
                             scheme-properties))
   (list
    (paragraph (style "fileblock_filetitle" (list* (alt-tag "div") (box-mode* "RfiletitleBox") scheme-properties))
               (list (make-element
                      (style "fileblock_filename" (list (css-style-addition custom-css)))
                      (if (string? filename)
                          (filepath filename)
                          filename))))
    (compound-paragraph 
     (style "fileblock_filecontent" (list* (alt-tag "div") (box-mode* "RfilecontentBox") scheme-properties))
     (decode-flow inside)))))









Added code-docs/tour.scrbl version [a54742ec].












































































































































1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

#lang scribble/manual

@; SPDX-License-Identifier: BlueOak-1.0.0
@; This file is licensed under the Blue Oak Model License 1.0.0.

@(require "scribble-helpers.rkt")

@(require (for-label racket/base pollen/core "../pollen.rkt"))

@title{How I Publish: A Quick Tour}

This isn’t a tutorial, since these steps probably won’t all work on your computer. Think of these
narrations like me talking while I drive.

@section{Creating an article}

Open a terminal window.

@terminal{@cmd{> cd /path/to/thelocalyarn}}

The @tt{make} command provides a high-level control panel for common tasks. Typing just make from
a terminal window shows a list of options:

@terminal{
@cmd{> make}
article     Start a new article from a template
help        Displays this help screen
publish     Sync all HTML and PDF stuff to the public web server
scribble    Rebuild code documentation and update Fossil repo
web         Rebuild all web content (not PDFs)
zap         Clear Pollen and Scribble cache, and remove all HTML output
}

Following the first option in this list, I type @tt{make article}, and enter a post title when
prompted:

@terminal{
@cmd{> make article}
racket -tm util/newpost.rkt
Enter title: @cmd{My New Post}
}

The script creates a new @filepath{.poly.pm} file using a normalized version of the title for the
filename, and opens it in my editor (this is currently hardcoded to use MacVim). When the file pops
up we see a basic template ready to edit:

@filebox["articles/my-new-post.poly.pm"
@codeblock|{
#lang pollen
 
◊; Copyright 2020 by Joel Dueck. All Rights Reserved.
◊(define-meta draft #t)
◊(define-meta published "2020-01-18")
 
◊title{My New Post}
 
Write here!
}|]

At this point I might delete the @tt{◊title} line, since specifying a formal title is optional
(other than the one needed to generate the filename). I might also add a @racket[define-meta] for
@tt{series} or @tt{topics}. 

As long as the @racket[define-meta] for @tt{draft} is @racket[#t], the new article will not appear
in the RSS feed, or in the blog or any series pages.

When satisfied with the post I’ll remove the @racket[define-meta] for @tt{draft}, save it one last
time, then go back to the terminal:

@terminal{
@cmd{> make web}
[lots of output: rebuilds blog pages, keyword index, RSS feed]
@cmd{> make publish}
[lots of output: uploads all web content to the server]
}

The article also needs to be added to the Fossil repository, so revisions to it will be tracked:

@terminal|{
|@cmd{> fossil add article/my-new-post.poly.pm}
ADDED  article/my-new-post.poly.pm

|@cmd{> fossil commit -m "Publish ‘My New Post’"}
Autosync:  https://joel@thelocalyarn.com/code
Round-trips: 2   Artifacts sent: 0  received: 1
Pull done, sent: 892  received: 8720  ip: 162.243.186.132
New_Version: 15507b62416716a3f0be3c444b0fc09aa4364b989140c5788cf679eb0b2463a6
Autosync:  https://joel@thelocalyarn.com/code
Round-trips: 2   Artifacts sent: 2  received: 1
Sync done, sent: 10153  received: 4680  ip: 162.243.186.132
}|

As you can see, Fossil does an automatic pull before the commit, and another automatic push
afterwards. This commit is now visible on the public timeline, and the source code for the article
can now be seen on the public repo at @tt{thelocalyarn.com/code/}.

@section{Adding notes to an article}

A few days (or years) after doing the above, I receive an email from Marjorie with commenting on
@italic{My New Post} and I decide to publish her comments.

I open the article in my editor and add some lines to the end:

@filebox["articles/my-new-post.poly.pm"
@codeblock|{
#lang pollen
 
◊; Copyright 2020 by Joel Dueck. All Rights Reserved.
◊(define-meta published "2020-01-18")
 
◊title{My New Post}
 
It’s a 4⨉4 treated. I got twenty, actually, for the backyard fence.

◊note[#:date "2020-01-23" #:author "Marjorie"]{
 Hope you sank that thing below the frost line.
}
}|]

This looks like a blog-style comment, but the @racket[note] tag function has some special powers
that typical comments don’t have, as we’ll see in a moment.

I save this, go back to the terminal and do @tt{make web} and @tt{make publish} as before.

Now if you open the article’s permlink, you’ll see the note appears in a “Further Notes” section at
the bottom — again, just like a normal blog post comment.

But if you go to the Blog section, you’ll see the note appearing in its own space right alongside
the other articles, as if it were a separate post. It will also appear in a separate entry in the
RSS feed.

@section{What’s not here yet}

Eventually there will be facilities for creating PDF files of individual articles, and print-ready
PDFs of books containing collections of articles. 









Modified makefile
from [9872e7c1]
to [ffa5dcb1].




︙


96
97
98
99
100
101
102

103
104
105
106
107
108
109

96
97
98
99
100
101
102
103
104
105
106
107
108
109
110








+








		--exclude=makefile 
	rm -rf ~/Desktop/publish
	fossil uv sync

scribble: ## Rebuild code documentation and update Fossil repo
	scribble --htmls +m --redirect https://docs.racket-lang.org/local-redirect/ code-docs/main.scrbl
	fossil uv rm scribbled/*
	mv scribbled/site-footer.html main/ || true
	rm -rf scribbled/*
	mv main/* scribbled/
	cp code-docs/scribble-iframe.html scribbled/scribble.html
	rm -rf main
	fossil uv add scribbled/*
	fossil uv sync





︙










This page was generated in about
0.013s by
Fossil 2.19 [318ab802db] 2022-05-27 02:05:21