◊(Local Yarn Code "Check-in [b4faafb8]")

Overview
Comment:Add listing schema to code docs
Timelines: family | ancestors | descendants | both | doc-expansion
Files: files | file ages | folders
SHA3-256: b4faafb89c5a4b6142301ac652a24766f10d82dde57e370294e9b50e74144ec1
User & Date: joel on 2020-02-19 07:24:14
Other Links: branch diff | manifest | tags
Context
2020-02-19
07:25
Merge doc updates. Closes [2e658da] check-in: 2cd87113 user: joel tags: trunk
07:24
Add listing schema to code docs Closed-Leaf check-in: b4faafb8 user: joel tags: doc-expansion
03:28
Further reorg code docs check-in: 82481872 user: joel tags: doc-expansion
Changes

Modified code-docs/cache.scrbl from [2e144f88] to [821a1577].

33
34
35
36
37
38
39

40
41
42
43
44
45
46


47

48
49
50
51
52
53
54
55
56
57

58
59
60
61
62
63
64
@section{Cache database}

@defthing[cache-conn connection?]{
The database connection.
}

@defproc[(init-cache-db!) void?]{

Creates and initializes the SQLite database cache file (named @filepath{vitreous.sqlite} and located
in the project root folder) by running queries to create tables in the database if they do not
exist.

This function is called automatically in @seclink["pollen-rkt"]{@filepath{pollen.rkt}} whenever HTML
is the target output. }



@defparam[current-plain-title title-plain non-empty-string? #:value "void"]{

Contains (or sets) the “plain” title (i.e., with no HTML markup) for the current article based on
analysis done by @racket[parse-and-cache-article!]. If the article did not specify a title,
a default title is supplied. If the article contained a @racket[note] that used the
@code{#:disposition} attribute, the disposition-mark may be included in the title.

This is a weird parameter, and at some point I will probably get rid of it and have
@racket[parse-and-cache-article!] supply it as an extra return value instead.

@margin-note{Note that this needs to be called @emph{after} @racket[parse-and-cache-article!] in
order to get an up-to-date value.}

}

@section{Retrieving cached data}

Some of this looks a little wacky, but it’s a case of putting a little extra complextity into the
back end to make things simple on the front end. These functions are most commonly used inside the
@emph{body} of a Pollen document (i.e., series pages). 







>





|

>
>

>










>







33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
@section{Cache database}

@defthing[cache-conn connection?]{
The database connection.
}

@defproc[(init-cache-db!) void?]{

Creates and initializes the SQLite database cache file (named @filepath{vitreous.sqlite} and located
in the project root folder) by running queries to create tables in the database if they do not
exist.

This function is called automatically in @seclink["pollen-rkt"]{@filepath{pollen.rkt}} whenever HTML
is the target output.

}

@defparam[current-plain-title title-plain non-empty-string? #:value "void"]{

Contains (or sets) the “plain” title (i.e., with no HTML markup) for the current article based on
analysis done by @racket[parse-and-cache-article!]. If the article did not specify a title,
a default title is supplied. If the article contained a @racket[note] that used the
@code{#:disposition} attribute, the disposition-mark may be included in the title.

This is a weird parameter, and at some point I will probably get rid of it and have
@racket[parse-and-cache-article!] supply it as an extra return value instead.

@margin-note{Note that this needs to be called @emph{after} @racket[parse-and-cache-article!] in
order to get an up-to-date value.}

}

@section{Retrieving cached data}

Some of this looks a little wacky, but it’s a case of putting a little extra complextity into the
back end to make things simple on the front end. These functions are most commonly used inside the
@emph{body} of a Pollen document (i.e., series pages). 
86
87
88
89
90
91
92

93
94
95
96
97
98
99
                                 [#:limit limit integer? -1]
                                 [order stringish? 'desc]) txexpr?]
              @defproc[(<listing-short>
                                 [query-func (-> any/c query?)]
                                 [#:series series (or/c string? (listof string?) boolean?) #t]
                                 [#:limit limit integer? -1]
                                 [order stringish? 'desc]) txexpr?])]{

Fetches the HTML for items from the SQLite cache, concatenates their HTML strings and returns
a @racket['style] tagged X-expression with this string as its element. The items will be ordered by
publish date according to @racket[_order] and optionally limited to the series specified in
@racket[_series].

The @racket[_query-func] should be either @racket[articles], which will create a listing of articles
only, or @racket[articles+notes], which will include notes intermingled with articles.







>







91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
                                 [#:limit limit integer? -1]
                                 [order stringish? 'desc]) txexpr?]
              @defproc[(<listing-short>
                                 [query-func (-> any/c query?)]
                                 [#:series series (or/c string? (listof string?) boolean?) #t]
                                 [#:limit limit integer? -1]
                                 [order stringish? 'desc]) txexpr?])]{

Fetches the HTML for items from the SQLite cache, concatenates their HTML strings and returns
a @racket['style] tagged X-expression with this string as its element. The items will be ordered by
publish date according to @racket[_order] and optionally limited to the series specified in
@racket[_series].

The @racket[_query-func] should be either @racket[articles], which will create a listing of articles
only, or @racket[articles+notes], which will include notes intermingled with articles.
115
116
117
118
119
120
121

122
123
124

125
126

127
128
129
130
131
132
133
134
135
136

137
138
139
140
141
142
143
144
145
146
147
148

149
150
151
152
153
154


155

156
157


158
159
160
161
162

163

164
165
166
167
168
169
170
evaluates to a negative integer (the default) is the same as having no limit.

The reason for enclosing the results in a @racket['style] txexpr is to prevent the HTML from being
escaped by @racket[->html] in the template. This tag was picked for the job because there will
generally never be a need to include any actual CSS information inside a @tt{<style>} tag in any
page, so it can be safely filtered out later. To remove the enclosing @tt{<style>} tag, see
@racket[unfence].

}

@defproc[(listing-htmls [listing-query query?]) (listof string?)]{

Returns the HTML bodies for the articles and/or notes returned by @racket[_listing-query] as a list
of strings.

}

@deftogether[(@defproc[(articles [type (or/c 'full 'excerpt 'short)]
                                 [#:series series (or/c string? (listof string?) boolean?) #t]
                                 [#:limit limit integer? -1]
                                 [order stringish? 'desc]) query?]
              @defproc[(articles+notes [type (or/c 'full 'excerpt 'short)]
                                       [#:series series (or/c string? (listof string?) boolean?) #t]
                                       [#:limit limit integer? -1]
                                       [order stringish? 'desc]) query?])]{

Create a query that fetches either articles only (@racket[articles]) or articles and notes
intermingled (@racket[articles+notes]), sorted by publish date and optionally limited to
a particular series.

Typically you will pass these functions by name to listing functions like @racket[<listing-full>]
rather than calling them directly.

@examples[#:eval example-eval
(articles 'full)]
}

@defproc[(unfence [html string?]) string?]{

Returns @racket[_html] with all occurrences of @racket["<style>"] and @racket["</style>"] removed.
The contents of the style tags are left intact.

Use this in templates with strings returned from @racket[->html] when called on docs that use the
@racket[<listing-full>] tag function or its siblings. }



@defproc[(series-grouped-list) (listof (listof cache:series?))]{

Return a list of lists of all @racket[cache:series] in the cache database. The series are grouped so
that series using the same value in the @tt{noun-plural} column appear together.}



@section{Deleting records}

@deftogether[(@defproc[(delete-article! [page stringish?]) void?]
              @defproc[(delete-notes!   [page stringish?]) void?])]{

Delete a particular article, or all notes for a particular article, respectively.

}

@section{Schema}

The cache database has four tables: @tt{articles}, @tt{notes}, @tt{index_entries} and @tt{series}. Each of these has a corresponding schema, shown below. In addition, there is a “virtual” schema, @tt{listing}, for use with queries which may or may not combine articles and notes intermingled.

The work of picking apart an article’s exported @tt{doc} and @tt{metas} into rows in these tables is done by @racket[parse-and-cache-article!].







>



>


>










>












>




|

>
>

>

|
>
>





>

>







121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
evaluates to a negative integer (the default) is the same as having no limit.

The reason for enclosing the results in a @racket['style] txexpr is to prevent the HTML from being
escaped by @racket[->html] in the template. This tag was picked for the job because there will
generally never be a need to include any actual CSS information inside a @tt{<style>} tag in any
page, so it can be safely filtered out later. To remove the enclosing @tt{<style>} tag, see
@racket[unfence].

}

@defproc[(listing-htmls [listing-query query?]) (listof string?)]{

Returns the HTML bodies for the articles and/or notes returned by @racket[_listing-query] as a list
of strings.

}

@deftogether[(@defproc[(articles [type (or/c 'full 'excerpt 'short)]
                                 [#:series series (or/c string? (listof string?) boolean?) #t]
                                 [#:limit limit integer? -1]
                                 [order stringish? 'desc]) query?]
              @defproc[(articles+notes [type (or/c 'full 'excerpt 'short)]
                                       [#:series series (or/c string? (listof string?) boolean?) #t]
                                       [#:limit limit integer? -1]
                                       [order stringish? 'desc]) query?])]{

Create a query that fetches either articles only (@racket[articles]) or articles and notes
intermingled (@racket[articles+notes]), sorted by publish date and optionally limited to
a particular series.

Typically you will pass these functions by name to listing functions like @racket[<listing-full>]
rather than calling them directly.

@examples[#:eval example-eval
(articles 'full)]
}

@defproc[(unfence [html string?]) string?]{

Returns @racket[_html] with all occurrences of @racket["<style>"] and @racket["</style>"] removed.
The contents of the style tags are left intact.

Use this in templates with strings returned from @racket[->html] when called on docs that use the
@racket[<listing-full>] tag function or its siblings.

}

@defproc[(series-grouped-list) (listof (listof cache:series?))]{

Return a list of lists of all @racket[cache:series] in the cache database. The series are grouped so
that series using the same value in the @tt{noun-plural} column appear together.

}

@section{Deleting records}

@deftogether[(@defproc[(delete-article! [page stringish?]) void?]
              @defproc[(delete-notes!   [page stringish?]) void?])]{
              
Delete a particular article, or all notes for a particular article, respectively.

}

@section{Schema}

The cache database has four tables: @tt{articles}, @tt{notes}, @tt{index_entries} and @tt{series}. Each of these has a corresponding schema, shown below. In addition, there is a “virtual” schema, @tt{listing}, for use with queries which may or may not combine articles and notes intermingled.

The work of picking apart an article’s exported @tt{doc} and @tt{metas} into rows in these tables is done by @racket[parse-and-cache-article!].
186
187
188
189
190
191
192

193

194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211

212

213
214
215
216
217
218
219
220
221

222

223
224
225
226
227
228
229
230

231
232











                           [doc-html             string/f]
                           [disposition          string/f]
                           [disp-html-anchor     string/f]
                           [listing-full-html    string/f]
                           [listing-excerpt-html string/f]
                           [listing-short-html   string/f])
            #:constructor-name make-cache:article]{

Table holding cached article information.

}

@defstruct*[cache:note ([id              id/f]
                        [page            symbol/f]
                        [html-anchor     string/f]
                        [title-html-flow string/f]
                        [title-plain     string/f]
                        [author          string/f]
                        [author-url      string/f]
                        [published       string/f]
                        [disposition     string/f]
                        [content-html    string/f]
                        [series-page     symbol/f]
                        [conceal         string/f]
                        [listing-full-html    string/f]
                        [listing-excerpt-html string/f]
                        [listing-short-html   string/f])
                       #:constructor-name make-cache:note]{

Table holding cached information on notes.

}

@defstruct*[cache:series ([id            id/f]
                          [page          symbol/f]
                          [title         string/f]
                          [published     string/f]
                          [noun-plural   string/f]
                          [noun-singular string/f])
                         #:constructor-name make-cache:series]{

Table holding cached information on series.

}

@defstruct*[cache:index-entry ([id id/f]
                               [entry       string/f]
                               [subentry    string/f]
                               [page        symbol/f]
                               [html-anchor string/f])
                              #:constructor-name make-cache:index-entry]{

Table holding cached information about index entries found in articles.
}


















>

>


















>

>









>

>








>

|
>
>
>
>
>
>
>
>
>
>
>
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
                           [doc-html             string/f]
                           [disposition          string/f]
                           [disp-html-anchor     string/f]
                           [listing-full-html    string/f]
                           [listing-excerpt-html string/f]
                           [listing-short-html   string/f])
            #:constructor-name make-cache:article]{

Table holding cached article information.

}

@defstruct*[cache:note ([id              id/f]
                        [page            symbol/f]
                        [html-anchor     string/f]
                        [title-html-flow string/f]
                        [title-plain     string/f]
                        [author          string/f]
                        [author-url      string/f]
                        [published       string/f]
                        [disposition     string/f]
                        [content-html    string/f]
                        [series-page     symbol/f]
                        [conceal         string/f]
                        [listing-full-html    string/f]
                        [listing-excerpt-html string/f]
                        [listing-short-html   string/f])
                       #:constructor-name make-cache:note]{

Table holding cached information on notes.

}

@defstruct*[cache:series ([id            id/f]
                          [page          symbol/f]
                          [title         string/f]
                          [published     string/f]
                          [noun-plural   string/f]
                          [noun-singular string/f])
                         #:constructor-name make-cache:series]{

Table holding cached information on series.

}

@defstruct*[cache:index-entry ([id id/f]
                               [entry       string/f]
                               [subentry    string/f]
                               [page        symbol/f]
                               [html-anchor string/f])
                              #:constructor-name make-cache:index-entry]{
                              
Table holding cached information about index entries found in articles.

}

@defstruct*[listing ([html        string/f]
                     [published   date/f]
                     [series-page symbol/f])
                    #:constructor-name make-listing]{

This is not a table that persists in the cache database; rather it is the schema targeted by
@racket[articles] and @racket[articles+notes] using deta’s @racket[project-onto].

}

Modified code-docs/main.scrbl from [e2e5a12a] to [662fc0af].

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
#lang scribble/manual

@; SPDX-License-Identifier: BlueOak-1.0.0
@; This file is licensed under the Blue Oak Model License 1.0.0.

@(require "scribble-helpers.rkt"
          racket/runtime-path
          (for-label racket/base
                     "../crystalize.rkt"))

@title{Local Yarn: source code notes}

@author{Joel Dueck}

These are my notes about the internals of the Local Yarn source code. In other words, a personal
reference, rather than a tutorial. These pages concern only the source code itself. Refer to the
wiki for info about deployment, etc.

You’ll get the most out of these notes if you have read @other-doc['(lib
"pollen/scribblings/pollen.scrbl")], and worked through the tutorials by hand.

@margin-note{Note that these pages are heavily interlinked with the central Racket documentation at
@tt{docs.racket-lang.org}, which are written and maintained by others. 

Some links from those pages will not work unless you @ext-link["#"]{open this page in its own tab}.
}











|




|
<


|







1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

17
18
19
20
21
22
23
24
25
26
#lang scribble/manual

@; SPDX-License-Identifier: BlueOak-1.0.0
@; This file is licensed under the Blue Oak Model License 1.0.0.

@(require "scribble-helpers.rkt"
          racket/runtime-path
          (for-label racket/base
                     "../crystalize.rkt"))

@title{Local Yarn: Source Code Notes}

@author{Joel Dueck}

These are my notes about the internals of the Local Yarn source code. In other words, a personal
reference, rather than a tutorial. 


You’ll get the most out of these notes if you have read @other-doc['(lib
"pollen/scribblings/pollen.scrbl")], and worked through the tutorials there by hand.

@margin-note{Note that these pages are heavily interlinked with the central Racket documentation at
@tt{docs.racket-lang.org}, which are written and maintained by others. 

Some links from those pages will not work unless you @ext-link["#"]{open this page in its own tab}.
}

54
55
56
57
58
59
60
61
@include-section["tour.scrbl"]
@include-section["design.scrbl"]
@include-section["pollen.scrbl"]  @; pollen.rkt
@include-section["dust.scrbl"]    @; dust.rkt
@include-section["snippets-html.scrbl"] @; you get the idea
@include-section["cache.scrbl"]
@include-section["crystalize.scrbl"]








<
53
54
55
56
57
58
59

@include-section["tour.scrbl"]
@include-section["design.scrbl"]
@include-section["pollen.scrbl"]  @; pollen.rkt
@include-section["dust.scrbl"]    @; dust.rkt
@include-section["snippets-html.scrbl"] @; you get the idea
@include-section["cache.scrbl"]
@include-section["crystalize.scrbl"]