DocRegExt.tex

\documentclass{ivoa}
\input tthdefs
\input gitmeta

\definecolor{termcolor}{rgb}{0.6,0.1,0.1}

\iftth
\def\vocterm#1{\emph{\color{termcolor}#1}}

\else
\def\vocterm{\startvocterm\realvocterm}
\def\realvocterm#1{\emph{\color{termcolor}#1}\endvocterm}
\begingroup
\gdef\breakablecolon{:\hskip0pt}
\catcode`\:=\active
\gdef\startvocterm{\begingroup
  \catcode`\:=\active\let:=\breakablecolon}
\gdef\endvocterm{\endgroup}
\endgroup
\fi

\usepackage{todonotes}
\lstloadlanguages{XML,sh,SQL}
\lstset{flexiblecolumns=true,tagstyle=\ttfamily, showstringspaces=False}

\ivoagroup{Registry}

\author{Demleitner, M.}
\author{Molinaro, M.}
\author{Iafrate, G.}
\author{Heinl, H.}

\editor{Demleitner, M.}

\previousversion[https://www.ivoa.net/documents/DocRegExt/20230426]{WD-1.0-20230426}
\previousversion[http://ivoa.net/documents/Notes/EDU/index.html]{Educational
Resources Note}


\title{Educational Resources in the Virtual Observatory}

\begin{document}

\begin{abstract}
Following the spirit of the Virtual Observatory (VO), the production and
provision of material to teach VO usage, both for introducing VO
workflows and as an element of a larger astronomy curriculum, should be
a joint, global, and distributed effort.  In order to ensure global
discoverability of the results of that effort, this standard defines an
extension to the VOResource metadata scheme suitable for educational
texts, and it describes how RegTAP can be used to then discover
educational material registered using that extension.

\end{abstract}


\section{Introduction}

The Virtual Observatory is a complex tool, introducing numerous
technologies not immediately familiar to the average astronomer.
In enabling university level students
and active researchers to fully exploit the VO's capabilities, course
materials and worked-out use-cases have been found to be an efficient means of
developing the necessary skills both in interactive course situations
like ``VO Days'' and beyond.
Efficient ways for interested users to locate such
material as well as for VO operators to curate it are highly desirable.

At the same time, advances in technology and
communications are creating new and exciting
opportunities for teachers to bring astronomy into their
classrooms.  As the VO makes science-grade data publicly available and
classroom sets of (suitably) networked computing devices are now
standard in schools, exciting projects come within
reach of teachers.  To realise these potentials,
it is necessary to disseminate educational material to help teachers
preparing classes.  Such material -- including documented step-by-step
tutorials and use cases explaining how to perform basic astrophysical research
using VO tools and resources -- exists in various formats and
has been translated in different languages.

Hence, both training professionals for VO exploitation and the provision
of educational material to school teachers and the greater public
require some way of handling metadata on the available material.  Since
the VO has no central body that could curate such a metadata collection
but instead uses a decentralised registry for the analogous problem of
service discovery, it appears natural to employ the VO Registry for the
management of text resource metadata as well.


\subsection{Previous Work}

In 2013, some providers of educational material began to address the
problem of registering educational texts with the goal of improving
their findability.

The VO at that time already had a registry extension for standards, which of
course are also text-like: StandardsRegExt \citep{2012ivoa.spec.0508H}.  This extension,
however, focuses on metadata important for standards – e.g.,
vocabularies and status – that is not pertinent for educational
material.  Conversely, it is not concerned with document language (which
can safely be assumed to be English for standards), and it disregards
the issue of locating formatted and source versions, which for educational
material is important.

Therefore, a new registry extension was designed, provisionally called
DocRegExt.  With perhaps a dozen records of that type in the Registry,
in 2016 a web page giving a near real-time display of the available
material, was created, VO Text Treasures, or
VOTT\footnote{\url{https://dc.g-vo.org/VOTT}} for short.  This extension was
published as part of the
IVOA note ``Educational Resources in the Virtual Observatory''
\citep{note:edumatters}.

Since then, the number of registered resources of the type
\xmlel{doc:Document} had risen to 42 as of March 2023.  It hence seems
advisable to give the registry extension a proper normative basis, which
is what this standard provides.


\subsection{Use Cases}

\label{sect:regext-usecases}

The design of DocRegExt has been guided by the
following discovery cases:


\begin{itemize}

\item Is there a tutorial covering discovering intermediate mass black
holes? (Standard VOResource is sufficient).

\item Is there a tutorial covering working with X-Ray data? (Standard
VOResource is sufficient).

\item Is there a tutorial dealing with planets suitable for school use?
(Standard VOResource is sufficient).

\item Is there a tutorial dealing with planets suitable for school use in
Italian? (That requires the declaration of the document language).

\item What are the subjects of maintained (in the sense of: probably
working in the VO as found by the students) tutorials?
(The active flag of standard VOResource is
unsuitable here since even outdated resources will still be accessible;
instead, this standard proposes to include the date the authors last
tried a tutorial using standard VOResource mechanisms).

\item Are there tutorials using redshifts? (This is addressed by allowing
table metadata in DocRegExt).

\item Where can I find an editable version of tutorial ivo://auth/tut1?
(This is solved by interfaces with a \xmlel{role} of \textit{source}).

\item Are there translations of tutorial ivo://auth/tut2? (This is
solved by modelling different translations as capabilities of the same
resource).

\item Is there material using service ivo://auth/svc1? (Declaring
standard VOResource relationships covers this use case).

\item Is there material about something visible tonight? (When tutorials
declare their \xmlel{coverage}, this is possible -- but few do that at this
point).

\item I found this VO tutorial somewhere on the net (``on a mirror'').  Is it
the latest version?  If not, where can I find an update? (Unless the
title of the text changed, standard VOResource should suffice).

\end{itemize}

An important additional use case is enabling attractive, browsable
lists of registred educational material.  In the operation of GAVO's
VOTT service mentioned above it was found that one
requirement resulting from this use case is direct access to formatted
material in order to enable thumbnail generation.

On the use cases of locating editable forms of such texts – which
has been found to be necessary fairly regularly – we note in passing
that representing source-product relationships is in principle in the
domain of provenance and thus not in the Registry's main scope. However, in
the case discussed here the relation is so simple and its representation
so useful that we propose to include it in a DocRegExt.

\subsection{Role within the VO Architecture}

\begin{figure}
\centering

\includegraphics[width=0.9\textwidth]{role_diagram.pdf}
\caption{Architecture diagram for this document}
\label{fig:archdiag}
\end{figure}

Fig.~\ref{fig:archdiag} shows the role this document plays within the
IVOA architecture \citep{2021ivoa.spec.1101D}.  It is closely related to
the following other standards:

\begin{itemize}
\item It extends the VO's basic resource metadata schema VOResource
\citep{2021ivoa.spec.1102D}.  In particular, we take certain liberties
with VOResource's notion of ``capability'' by re-using it for the
different translations of a document.

\item We give discovery recipes in terms of the Registry access protocol
RegTAP \citep{2019ivoa.spec.1011D}.

\item The base type of the \xmlel{Document} resource type defined here
is \xmlel{vs:Catalog\-Resource} from VODataService
\citep{2021ivoa.spec.1102D}.  While that type is really intended to
describe astronomical \emph{data} (rather than material using such
data), pieces of metadata like the tableset or the coverage have clear
applications within the current context.  With the moderate extension of
the notion of a capability mentioned above, the match in the concepts is
surprisingly smooth.

\end{itemize}

\section{The DocRegExt Schema}

\label{sect:regext}

To satisfy the requirements derived above, we have designed a registry extension with
two definitions.
To avoid unnecessary incompatibilities when migrating to a proper IVOA
standard, we use the namespace URI
$$\hbox{\nolinkurl{http://www.ivoa.net/xml/DocRegExt/v1}}$$
for DocRegExt even while the schema cannot actually be retrieved from
there.

The canonical schema prefix for DocRegExt is \texttt{doc}.

To let authors define comprehensive metadata, the schema
re-uses the \xmlel{vs:Ca\-talogResource} type
from VODataService 1.2 \citep{2021ivoa.spec.1102D} to construct
the \xmlel{doc:Document} resource type.  We import the schema by major
version, such that further enhancements within the VODataService version
1 series will apply to DocRegExt records, too.

While the schema does not limit what kinds of capabilities a
\xmlel{doc:Document} record has -- it is conceivable that custom
services for use in a particular tutorial
are communicated in this way --, access to actual files is
enabled using \xmlel{doc:Edition}-typed capabilities.  It may be
argued that this use of VOResource capabilities stretches their
semantics a bit.  We argue, however, that these documents can well be
understood as parameterless service endpoints.  Using capabilities
furthermore allows a complete representation of the metadata in RegTAP
without any extra tables (cf.~sect.~\ref{sect:docregext-regtap}).

The resource-level \vorent{referenceURL} in \xmlel{doc:Document} records should
be some sort of landing page with an abstract of the text and links to
the full texts and perhaps the document source(s).  When using the
versioned repository (sect.~\ref{sect:svn-repo}), this could be the
top-level README file within the Version Control System.  For simple documents, it is
acceptable to use the English-language document itself as
\xmlel{referenceURL}; documents only available in non-English should
provide a landing page with an English-language abstract, though.

The \vorent{facility} and \vorent{instrument} items should only be set
if the text in question actually exploits particular properties of the
concrete instrument.  A \vorent{tableset} can be given for the central
table-like structures a text deals with and facilitates discovery by
physics via the UCDs given in the tableset.

Document-typed resource records should define relations to other
general resources (e.g., applications or services)
they use. The IVOA vocabulary
relationship\_type\footnote{\url{https://www.ivoa.net/rdf/voresource/relationship_type}}
defines what terms are legal in the declaration of relationships.
Document records should preferably use \vocterm{Cites} and
in particular declare relationships to tools.  For VO-registered
services or standards, these relationships should provide the services'
IVOA identifiers.
Tutorials
specifically introducing one or more services should define
\vocterm{IsSupplementTo} relationships to these services.

Client-side software or libraries do not have Registry records.  To
reliably identify software, DocRegExt recommends the use of
ASCL\footnote{\url{https://www.ascl.net}} identifiers.  These should be
used in the \verb|relatedResource| element's \xmlel{altIdentifier}
attribute.  For instance, to declare that a tutorial uses the Cassis
client, it would say:

\begin{lstlisting}[language=XML,basicstyle=\footnotesize]
<relationship>
  <relationshipType>Cites</relationshipType>
  <relatedResource>Specflow</relatedResource>
  <relatedResource altIdentifier="ascl:1402.013"
    >Cassis</relatedResource>
</relationship>
\end{lstlisting}

Document-typed resource records should contain a \vorent{date} element
with a \vorent{role} of \emph{Inspected}.  This should correspond to the
last time that the current editor has made sure the tutorial or use case
works as expected.  See the date\_role
vocabulary\footnote{\url{http://www.ivoa.net/rdf/voresource/date_role}}
for details.

Each \xmlel{doc:Edition}-typed capability should
correspond to a translation of the document.  It
is recommended to list the English-language version first if it exists.

The following description of the \xmlel{doc:Edition} capability
is generated from the schema file.

% GENERATED: !schemadoc DocRegExt-v1.0.xsd Edition
\begin{generated}
\begingroup
        \renewcommand*\descriptionlabel[1]{%
        \hbox to 5.5em{\emph{#1}\hfil}}\vspace{2ex}\noindent\textbf{\xmlel{doc:Edition} Type Schema Documentation}

\noindent{\small
        An “edition” (typically: translation) of the document.
      \par}

\noindent{\small
        Although for a while, multiple editions of the document in one language
        may be given (corresponding perhaps to two “major” versions), in
        general, only the latest version of the document per language should be
        present.

        At least one interface with
        role={"}rendered{"} must be present, typcially of type doc:StaticFile.
        The access URL of the interface
        points to a rendered version of the edition (preferably in PDF,
        but other formats like HTML or Python notebooks are acceptable, too).

        Editors are strongly encouraged to also provide an
        interface with role={"}source{"}, the accessURL of which should point
        to an editable version of the document, a version controlled
        repository, or the like.
      \par}

\vspace{1ex}\noindent\textbf{\xmlel{doc:Edition} Type Schema Definition}

\begin{lstlisting}[language=XML,basicstyle=\footnotesize]
<xs:complexType name="Edition" >
  <xs:complexContent >
    <xs:extension base="vr:Capability" >
      <xs:sequence >
        <xs:element name="languageCode" type="xs:token" minOccurs="1"
                  maxOccurs="1" />
        <xs:element name="locTitle" type="xs:token" minOccurs="0"
                  maxOccurs="1" />
      </xs:sequence>
    </xs:extension>
  </xs:complexContent>
</xs:complexType>
\end{lstlisting}

\vspace{0.5ex}\noindent\textbf{\xmlel{doc:Edition} Extension Metadata Elements}

\begingroup\small\begin{bigdescription}\item[Element \xmlel{languageCode}]
\begin{description}
\item[Type] string: \xmlel{xs:token}
\item[Meaning]
                The language this document is (mainly) written in,
                as an BCP 47 language code.

\item[Occurrence] required
\item[Comment]
                The country codes must be given in all lowercase.  This
                results in strings like en, de, or, where regions
                might actually matter, es-es or es-mx.

                This language is also the language for locTitle,
                irrespective of that element's xml:lang setting.


\end{description}
\item[Element \xmlel{locTitle}]
\begin{description}
\item[Type] string: \xmlel{xs:token}
\item[Meaning]
                The translated document's title in the language specified
                by the language sibling.

\item[Occurrence] optional
\item[Comment]
                It is recommended to set this element's xml:lang element
                to the same value as the language sibling.  The reason
                xml:lang is not used in the first place is that with an
                extra element, enforcing that the content language is
                given is more straightforward.


\end{description}


\end{bigdescription}\endgroup

\endgroup
\end{generated}

% /GENERATED

Interface element(s) within \xmlel{doc:Edition}-typed capabilities give
access to the actual document(s) comprising the text.  At least one
interface must have a role \verb|rendered|, which should yield the main
text.  If at all possible a link to the source form should be provided
in an interface with the role \verb|source|.

For the rendered document, publishers should directly link to the
consumable artefact (e.g., a PDF) rather than, say, a landing page
(which would be linked to in \xmlel{referenceURL}).  In that case,
resource records should use interface elements of
\xmlel{doc:StaticFile}.  When the linked document is not a PDF, they
should also give that type's \xmlel{resultType} child as appropriate.
This specification comes with an example
record\footnote{\auxiliaryurl{notebook-example.xml}} showing how this
would look like for a Python notebook.

% GENERATED: !schemadoc DocRegExt-v1.0.xsd StaticFile
\begin{generated}
\begingroup
        \renewcommand*\descriptionlabel[1]{%
        \hbox to 5.5em{\emph{#1}\hfil}}\vspace{2ex}\noindent\textbf{\xmlel{doc:StaticFile} Type Schema Documentation}

\noindent{\small
        An endpoint that returns a single, constant document.
      \par}

\noindent{\small
        Data providers are encouraged to provide a media type for
        that document.  For archives containing multiple files, multiple media
        types can be given.  Endpoints requiring further interaction
        before the desired document is retrieved (e.g., landing pages
        or version control interfaces) should use the vr:WebBrowser type.
      \par}

\vspace{1ex}\noindent\textbf{\xmlel{doc:StaticFile} Type Schema Definition}

\begin{lstlisting}[language=XML,basicstyle=\footnotesize]
<xs:complexType name="StaticFile" >
  <xs:complexContent >
    <xs:extension base="vr:Interface" >
      <xs:sequence >
        <xs:element name="resultType" type="xs:string" minOccurs="0"
                  maxOccurs="unbounded" />
      </xs:sequence>
    </xs:extension>
  </xs:complexContent>
</xs:complexType>
\end{lstlisting}

\vspace{0.5ex}\noindent\textbf{\xmlel{doc:StaticFile} Extension Metadata Elements}

\begingroup\small\begin{bigdescription}\item[Element \xmlel{resultType}]
\begin{description}
\item[Type] string: \xmlel{xs:string}
\item[Meaning]
                An RFC 6838 media type of the document returned, or of
                a component of the document for archives.

\item[Occurrence] optional; multiple occurrences allowed.

\end{description}


\end{bigdescription}\endgroup

\endgroup
\end{generated}

% /GENERATED


\section{Locating Texts using RegTAP}
\label{sect:docregext-regtap}

In the relational registry \citep{2019ivoa.spec.1011D}, DocRegExt is
straightforwardly represented in the standard VOResource tables.  In
particular, to find all access URLs for documents together with their
English-language titles, one would write:

\begin{lstlisting}[language=SQL]
SELECT res_title, access_url FROM
  rr.resource
  NATURAL JOIN rr.interface
WHERE
  res_type='doc:document'
  and intf_role='rendered'
\end{lstlisting}

The \xmlel{languageCode} and \xmlel{locTitle} elements from the
\xmlel{doc:Edition} capability extension are mapped into
\verb|res_details| with the following \verb|detail_xpath|s:

\begin{itemize}

\item \texttt{/capability/languageCode} -- the document language as an
BCP 47 \citep{std:BCP47} language code.
\item \texttt{/capability/locTitle} -- the title in the national
language.
\end{itemize}

The \xmlel{resultType} child of \xmlel{doc:StaticFile} is mapped to
the \verb|result_type| column of RegTAP's \verb|interface| table.

To locate texts defining relationships to specific software tools, use
the relationship table's alt\_identifier column with an identifier
obtained from ascl.net:

\begin{lstlisting}[language=SQL]
SELECT ivoid, res_title, related_name FROM
  rr.resource
  NATURAL JOIN rr.relationship
WHERE
  res_type='doc:document'
  AND related_alt_identifier='ascl:1112.019'
\end{lstlisting}

Material that comes with a Python notebook and the URIs of these
notebooks can be discovered with

\begin{lstlisting}[language=SQL]
SELECT ivoid, res_title, access_url FROM
  rr.resource
  NATURAL JOIN rr.interface
WHERE
  res_type='doc:document'
  AND result_type='application/x-ipynb+json'
\end{lstlisting}

The downside of not defining an extra table for the documents is that
certain query patterns become somewhat clumsy.  For instance, to list
the English and Italian titles of all texts available in Italian, one
has to carefully join two subqueries to \verb|res_details|:

\begin{lstlisting}[language=SQL]
SELECT res_title, loctitle FROM
  rr.resource
  NATURAL JOIN (
    SELECT ivoid, loctitle FROM (
        SELECT ivoid, cap_index, detail_value as loctitle
        FROM rr.res_detail
        WHERE detail_xpath='/capability/locTitle') AS titles
      NATURAL JOIN (
        SELECT ivoid, cap_index
        FROM rr.res_detail
        WHERE
          detail_xpath='/capability/languageCode'
          AND detail_value LIKE 'it%') AS italiancaps
    ) as loctitles
WHERE
  res_type='doc:document'
\end{lstlisting}


\appendix

\section{An Example DocRegExt Record}

This document is accompanied with three example records showcasing most of
DocRegExt's features.

One example describes a tutorial for determining the distance to the Crab
nebula\footnote{\auxiliaryurl{m1distance-example.xml}} and shows
a resource with multiple translations.
Note that no interfaces with \textit{source} roles
are given for the German and Spanish versions.  This is because the
corresponding files were lost (see sect.~\ref{sect:svn-repo} for a
proposal designed to reduce such problems).

The second example describes a use case for a particular set of
low-resolution spectra\footnote{\auxiliaryurl{dfbs-example.xml}} and is
thus related to special services having particular tables.  This is
indicated using related resources and by including a tableset describing
these tables.

The last example\footnote{\auxiliaryurl{notebook-example.xml}} shows how
to properly declare that a tutorial comes in the form of a python
notebook and declares a diverse set of relationships to other resources.

\section{A versioned repository for tutorials}

\label{sect:svn-repo}

Registering text documents as VO resources allows searching for tutorials
and similar
material through standard registry interfaces, but keeping
tutorials up to date, in their master form and also in their translated
versions, is an obviously important management issue not really
addressed by the Registry.

For tracking changes and versions, the standard tool is a version
control system.  Therefore,
a versioned repository (using subversion as the version control system)
has been set up at GAVO data
centre\footnote{\url{http://svn.ari.uni-heidelberg.de/svn/edu/}}.
It collects some of the
already existing VO tutorials with the goal of preserving them and
letting users
update and translate them.

The repository has an internal structure designed to enable:

\begin{itemize}

\item different national languages (master language set to english){}

\item translation vs.~master language updates{}

\item licensing, in order to clarify how and whether a tutorial can be changed or re-used{}

\item additional material used by tutorials

\item access roles to let everyone access tutorials but prevent untrusted updates or additions to it

\end{itemize}

Details of this structure are discussed in a \texttt{README} file at the
root of the
repository\footnote{\url{http://svn.ari.uni-heidelberg.de/svn/edu/trunk/README}}.

The repository is intended to work as an archive and a space for cooperative
development of educational material for the VO.  A migration to some
git-based platform is under consideration.

Within the repository, there is also a relatively lightweight style file
with several definitions useful for tutorials.  Authors starting new
tutorials are encouraged to consider using it.  For a short introduction
to it, see its README\footnote{\url{http://svn.ari.uni-heidelberg.de/svn/edu/trunk/latex-votut/README}}.



\section{Changes}

\subsection{Changes from PR-1.0-2025-11-02}

\begin{itemize}
\item Recommending the usage of ASCL identifiers to declare
relationships to software.
\end{itemize}

\subsection{Changes from WD-1.0-20230426}

None.

\subsection{Changes from the Note}

\begin{itemize}
\item Following XML itself, we now ask for BCP 47 rather than RFC
3066 language codes.
\item Now basing doc:Document on vs:CatalogResource.
\item Removed the section on the ``educational registry''.
\end{itemize}

\bibliography{ivoatex/ivoabib,ivoatex/docrepo}

\end{document}