[doc] more reorganization in README.md and other pages

Author: knopers8

README.md

@@ -2,63 +2,167 @@
 [![godoc](https://img.shields.io/badge/godoc-Reference-5272B4.svg)](https://godoc.org/github.com/AliceO2Group/Control)
 # AliECS
 
-The ALICE Experiment Control System (**AliECS**) is the piece of software to drive and control the data taking in the ALICE experiment at CERN.
+The ALICE Experiment Control System (**AliECS**) is the piece of software to drive and control data taking activities in the experiment.
 It is a distributed system that combines state of the art cluster resource management and experiment control functionalities into a single comprehensive solution.
+
 Please refer to the [CHEP 2023 paper](https://doi.org/10.1051/epjconf/202429502027) for the latest design overview.
 
 ## How to get started
 
-Regardless of your use case, it is recommended to get acquainted with the main [AliECS concepts](docs/handbook/concepts.md).
+Regardless of your particular interests, it is recommended to get acquainted with the main [AliECS concepts](docs/handbook/concepts.md).
 
 After that, please find your concrete use case:
 
-* I want to **run AliECS** and other O²/FLP software
+### I want to **run AliECS** and other O²/FLP software
 
-    :arrow_right: [O²/FLP Suite deployment instructions](https://alice-flp.docs.cern.ch/system-configuration/utils/o2-flp-setup/)
+:arrow_right: [O²/FLP Suite deployment instructions](https://alice-flp.docs.cern.ch/system-configuration/utils/o2-flp-setup/)
 
-    These instructions apply to both single-node and multi-node deployments.
-    Contact [alice-o2-flp-support](mailto:alice-o2-flp-support@cern.ch) for assistance with provisioning and deployment.
+These instructions apply to both single-node and multi-node deployments.
+Contact [alice-o2-flp-support](mailto:alice-o2-flp-support@cern.ch) for assistance with provisioning and deployment.
 
-    There are two ways of interacting with AliECS:
+There are two ways of interacting with AliECS:
 
-        * The AliECS GUI (a.k.a. Control GUI, COG) - not in this repository, but included in most deployments, recommended
+- The AliECS GUI (a.k.a. Control GUI, COG) - not in this repository, but included in most deployments, recommended
 
-            :arrow_right: [AliECS GUI documentation](hacking/COG.md)
+  :arrow_right: [AliECS GUI documentation](hacking/COG.md)
 
-        * `coconut` - the command-line control and configuration utility, included with AliECS core, typically for developers and advanced users
+- `coconut` - the command-line control and configuration utility, included with AliECS core, typically for developers and advanced users
 
-            :arrow_right: [Using `coconut`](https://alice-flp.docs.cern.ch/aliecs/coconut/)
+  :arrow_right: [Using `coconut`](https://alice-flp.docs.cern.ch/aliecs/coconut/)
 
-            :arrow_right: [`coconut` command reference](https://alice-flp.docs.cern.ch/aliecs/coconut/doc/coconut/)
+  :arrow_right: [`coconut` command reference](https://alice-flp.docs.cern.ch/aliecs/coconut/doc/coconut/)
 
-* I want to ensure AliECS can **run and control my process**
+### I want to ensure AliECS can **run and control my process**
 
-    * My software is based on FairMQ and/or O² DPL (Data Processing Later)
-    
-        :palm_tree: AliECS natively supports FairMQ (and DPL) devices.
-        Head to [ControlWorkflows](https://github.com/AliceO2Group/ControlWorkflows) for instructions on how to configure your software to be controlled by AliECS.
-    
-    * My software does not use FairMQ and/or DPL, but should be controlled through a state machine
-    
-        :telescope: See [the OCC documentation](occ/README.md) to learn how to integrate the O² Control and Configuration library with your software. [Readout](https://github.com/AliceO2Group/Readout) is an example of this setup.
-        Once ready, head to [ControlWorkflows](https://github.com/AliceO2Group/ControlWorkflows) for instructions on how to configure it to be controlled by AliECS.
+* **My software is based on FairMQ and/or O² DPL (Data Processing Later)**
 
-    * My software is a command line utility with no state machine
-    
-        :palm_tree: AliECS natively supports generic commands.
-        Head to [ControlWorkflows](https://github.com/AliceO2Group/ControlWorkflows) for instructions to have your command ran by AliECS.
-        Make sure the task template for your command sets the control mode to `basic` ([see example](https://github.com/AliceO2Group/ControlWorkflows/blob/master/tasks/o2-roc-cleanup.yaml)).
-    
-* I want to **develop** AliECS
+    AliECS natively supports FairMQ (and DPL) devices.
+    Head to [ControlWorkflows](https://github.com/AliceO2Group/ControlWorkflows) for instructions on how to configure your software to be controlled by AliECS.
+  
+* **My software does not use FairMQ and/or DPL, but should be controlled through a state machine**
+  
+    See [the OCC documentation](occ/README.md) to learn how to integrate the O² Control and Configuration library with your software. [Readout](https://github.com/AliceO2Group/Readout) is an example of this setup.
 
-    :hammer_and_wrench: [Building instructions](https://alice-flp.docs.cern.ch/aliecs/building/)
-    
-    :arrow_right: [Running instructions](https://alice-flp.docs.cern.ch/aliecs/running/)
+    Once ready, head to [ControlWorkflows](https://github.com/AliceO2Group/ControlWorkflows) for instructions on how to configure it to be controlled by AliECS.
 
-* I want my service to communicate with AliECS upon environment state transitions
+* **My software is a command line utility with no state machine**
+  
+    AliECS natively supports generic commands.
+    Head to [ControlWorkflows](https://github.com/AliceO2Group/ControlWorkflows) for instructions to have your command ran by AliECS.
+    Make sure the task template for your command sets the control mode to `basic` ([see example](https://github.com/AliceO2Group/ControlWorkflows/blob/master/tasks/o2-roc-cleanup.yaml)).
+    
+### I want to develop AliECS
 
-  * Learn more about the [plugin system](TODO)
+:hammer_and_wrench: [Building instructions](https://alice-flp.docs.cern.ch/aliecs/building/)
+    
+:arrow_right: [Running instructions](https://alice-flp.docs.cern.ch/aliecs/running/)
+
+TODO CONTRIBUTING.md
+
+### I want to receive updates about environments or services controlled by AliECS
+
+:book: Learn more about the [kafka event service](docs/kafka.md)
+
+### I want my service to communicate with AliECS
+
+:book: Learn more about the [plugin system](TODO)
+
+## Table of Contents
+
+* Introduction
+  * [Basic Concepts](/docs/handbook/concepts.md#basic-concepts)
+    * [Tasks](/docs/handbook/concepts.md#tasks)
+    * [Workflows, roles and environments](/docs/handbook/concepts.md#workflows-roles-and-environments)
+  * [Design Overview](/docs/handbook/overview.md#design-overview)
+    * [AliECS Structure](/docs/handbook/overview.md#aliecs-structure)
+    * [Resource Management](/docs/handbook/overview.md#resource-management)
+    * [FairMQ](/docs/handbook/overview.md#fairmq)
+    * [State machines](/docs/handbook/overview.md#state-machines)
+
+* Component reference
+  * [AliECS GUI](/hacking/cog.md)
+  * AliECS core
+    * [Workflow Configuration](/docs/handbook/configuration.md#workflow-configuration)
+      * [The AliECS workflow template language](/docs/handbook/configuration.md#the-aliecs-workflow-template-language)
+      * [Workflow template structure](/docs/handbook/configuration.md#workflow-template-structure)
+        * [Task roles](/docs/handbook/configuration.md#task-roles)
+        * [Call roles](/docs/handbook/configuration.md#call-roles)
+        * [Aggregator roles](/docs/handbook/configuration.md#aggregator-roles)
+        * [Iterator roles](/docs/handbook/configuration.md#iterator-roles)
+        * [Include roles](/docs/handbook/configuration.md#include-roles)
+        * [Template expressions](/docs/handbook/configuration.md#template-expressions)
+    * [Task Configuration](/docs/handbook/configuration.md#task-configuration)
+      * [Task template structure](/docs/handbook/configuration.md#task-template-structure)
+      * [Variables pushed to controlled tasks](/docs/handbook/configuration.md#variables-pushed-to-controlled-tasks)
+      * [Resource wants and limits](/docs/handbook/configuration.md#resource-wants-and-limits)
+    * plugin_system.md TODO
+    * [Environment operation order](/docs/handbook/operation_order.md#environment-operation-order)
+      * [START_ACTIVITY (Start Of Run)](/docs/handbook/operation_order.md#start_activity-start-of-run)
+      * [STOP_ACTIVITY (End Of Run)](/docs/handbook/operation_order.md#stop_activity-end-of-run)
+    * [Integrated service operations](/docs/handbook/operation_order.md#integrated-service-operations)
+      * [DCS](/docs/handbook/operation_order.md#dcs)
+        * [DCS operations](/docs/handbook/operation_order.md#dcs-operations)
+        * [DCS PrepareForRun behaviour](/docs/handbook/operation_order.md#dcs-prepareforrun-behaviour)
+        * [DCS StartOfRun behaviour](/docs/handbook/operation_order.md#dcs-startofrun-behaviour)
+        * [DCS EndOfRun behaviour](/docs/handbook/operation_order.md#dcs-endofrun-behaviour)
+    * [Protocol documentation](/docs/apidocs_aliecs.md)
+  * coconut
+    * [Configuration file](/coconut/README.md#configuration-file)
+    * [Using coconut](/coconut/README.md#using-coconut)
+      * [Creating an environment](/coconut/README.md#creating-an-environment)
+      * [Controlling an environment](/coconut/README.md#controlling-an-environment)
+    * [Command reference](/coconut/doc/coconut.md)
+  * apricot
+    * [Apricot overview](/apricot/README.md)
+    * [HTTP service](/apricot/docs/apricot_http_service.md#apricot-http-service)
+      * [Configuration](/apricot/docs/apricot_http_service.md#configuration)
+      * [Usage and options](/apricot/docs/apricot_http_service.md#usage-and-options)
+      * [Examples](/apricot/docs/apricot_http_service.md#examples)
+    * [Protocol documentation](/docs/apidocs_apricot.md)
+    * [Command reference](/apricot/docs/apricot.md) 
+  * occ
+    * [O² Control and Configuration Components](/occ/README.md#o-control-and-configuration-components)
+      * [Developer quick start instructions for OCClib](/occ/README.md#developer-quick-start-instructions-for-occlib)
+      * [Manual build instructions](/occ/README.md#manual-build-instructions)
+      * [Run example](/occ/README.md#run-example)
+      * [The OCC state machine](/occ/README.md#the-occ-state-machine)
+      * [Single process control with peanut](/occ/README.md#single-process-control-with-peanut)
+      * [OCC API debugging with grpcc](/occ/README.md#occ-api-debugging-with-grpcc)
+    * [Dummy process example for OCC library](/occ/occlib/examples/dummy-process/README.md#dummy-process-example-for-occ-library)
+    * [Protocol documentation](/docs/apidocs_occ.md)
+  * [peanut](/occ/peanut/README.md)
+  * Event service
+    * [Kafka producer functionality in AliECS core](/docs/kafka.md#kafka-producer-functionality-in-aliecs-core)
+      * [Making sure that AliECS sends messages](/docs/kafka.md#making-sure-that-aliecs-sends-messages)
+      * [Currently available topics](/docs/kafka.md#currently-available-topics)
+      * [Decoding the messages](/docs/kafka.md#decoding-the-messages)
+    * [Legacy events: Kafka plugin](/docs/kafka.md#legacy-events-kafka-plugin)
+      * [Making sure that AliECS sends messages](/docs/kafka.md#making-sure-that-aliecs-sends-messages-1)
+      * [Currently available topics](/docs/kafka.md#currently-available-topics-1)
+      * [Decoding the messages](/docs/kafka.md#decoding-the-messages-1)
+      * [Getting Start of Run and End of Run notifications](/docs/kafka.md#getting-start-of-run-and-end-of-run-notifications)
+      * [Using Kafka debug tools](/docs/kafka.md#using-kafka-debug-tools)
+
+* Developer documentation
+  * [AliECS pkg.go.dev documentation](https://pkg.go.dev/github.com/AliceO2Group/Control)
+  * [Building AliECS](/docs/building.md#building-aliecs)
+    * [Overview](/docs/building.md#overview)
+    * [Building with aliBuild](/docs/building.md#building-with-alibuild)
+    * [Manual build](/docs/building.md#manual-build)
+      * [Go environment](/docs/building.md#go-environment)
+      * [Clone and build (Go components only)](/docs/building.md#clone-and-build-go-components-only)
+  * [Makefile reference](/docs/makefile_reference.md)
+  * [Component Configuration](/docs/handbook/appconfiguration.md#component-configuration)
+    * [Connectivity to controlled nodes](/docs/handbook/appconfiguration.md#connectivity-to-controlled-nodes)
+  * [Running AliECS as a developer](/docs/running.md#running-aliecs-as-a-developer)
+    * [Running the AliECS core](/docs/running.md#running-the-aliecs-core)
+  * [Running AliECS in production](/docs/running.md#running-aliecs-in-production)
+    * [Health checks](/docs/running.md#health-checks)
+  * [Development Information](/docs/development.md#development-information)
+    * [Release Procedure](/docs/development.md#release-procedure)
+  * [OCC API debugging with grpcc](/docs/using_grpcc_occ.md#occ-api-debugging-with-grpcc)
+  * CONTRIBUTING.md TODO
+
+* Resources
+  * T. Mrnjavac et. al, [AliECS: A New Experiment Control System for the ALICE Experiment](https://doi.org/10.1051/epjconf/202429502027), CHEP23
 
-* I want to receive updates about environments or services controlled by AliECS
- 
-    * [Receive events published by AliECS via Kafka](docs/kafka.md)

apricot/README.md

@@ -1,14 +1,9 @@
 # `APRICOT`
 
-**A** **p**rocessor and **r**epos**i**tory for **co**nfiguration **t**emplates
+TODO write 
 
-The `o2-apricot` binary implements a centralized configuration (micro)service for ALICE O².
 
-```
-Usage of bin/o2-apricot:
-      --backendUri string   URI of the Consul server or YAML configuration file (default "consul://127.0.0.1:8500")
-      --listenPort int      Port of apricot server (default 32101)
-      --verbose             Verbose logging
-```
+### SEE ALSO
 
-Protofile: [apricot.proto](protos/apricot.proto)
+* [apricot HTTP service](docs/apricot_http_service.md) - make essential cluster information available via a web server
+* Protofile: [apricot.proto](protos/apricot.proto)

docs/faq.md

@@ -2,10 +2,17 @@
 
 From a logical point of view of data processing deployment and control, AliECS deals with concepts such as **environments**, **roles** and **tasks**, the understanding of which is paramount for using AliECS effectively.
 
+## Tasks
+
 The basic unit of scheduling in AliECS is a **task**. A task generally corresponds to a process. Sometimes this is a process that can receive and respond to OCC-compatible control messages (also called a **stateful task**), and other times this is simply a shell script or command line tool invocation (also called a **stateless task** or **basic task**).
 
+## Workflows, roles and environments
+
 All AliECS **workflows** are collections of tasks, which together form a coherent data processing chain.
 
-Tasks are the leaves in a tree of roles. A **role** is a runtime subdivision of the complete system, it represents a kind of operation along with its resources (but less than a complete data processing chain). Each task implements one or more roles. Roles allow binding tasks or groups of tasks to specific host attributes, detectors and configuration values. Each role represents either a single task, or a group of child roles. If tasks are leaves, roles are all the other nodes in the control tree of an environment.
+Tasks are the leaves in a tree of roles. A **role** is a runtime subdivision of the complete system, it represents a kind of operation along with its resources (but less than a complete data processing chain). Each task implements one or more roles. Roles allow binding tasks or groups of tasks to specific host attributes, detectors and configuration values. Each role represents either a single task, or a group of child roles. While tasks are leaves, roles are all the other nodes in the control tree of an environment.
 
 These novel, more flexible and more easily deployable abstractions represent the evolution of Run 2 abstractions such as ECS partitions. In memory, a tree of O² roles, along with their tasks and their configuration is a **workflow**. A workflow aggregates the collective state of its constituent O2 roles. A running workflow, along with associated detectors and other hardware and software resources required for experiment operation constitutes an **environment**.
+
+
+TODO add a diagram for the above, add hooks, activity/run?

docs/handbook/concepts.md

@@ -167,21 +167,6 @@ roles:
 
 In the absence of an explicit `critical` trait for a given task role, the assumed default value is `critical: true`.
 
-#### State machine callbacks moments
-
-The underlying state machine library allows us to add callbacks upon entering and leaving states as well as before and after events (transitions).
-This is the order of callback execution upon a state transition:
-1. `before_<EVENT>` - called before event named `<EVENT>`
-2. `before_event` - called before all events
-3. `leave_<OLD_STATE>` - called before leaving `<OLD_STATE>`
-4. `leave_state` - called before leaving all states
-5. `enter_<NEW_STATE>`, `<NEW_STATE>` - called after entering `<NEW_STATE>`
-6. `enter_state` - called after entering all states
-7. `after_<EVENT>`, `<EVENT>` - called after event named `<EVENT>`
-8. `after_event` - called after all events
-
-Callback execution is further refined with integer indexes, with the syntax `±index`, e.g. `before_CONFIGURE+2`, `enter_CONFIGURED-666`. An expression with no index is assumed to be indexed `+0`. These indexes do not correspond to timestamps, they are discrete labels that allow more granularity in callbacks, ensuring a strict ordering of callback opportunities within a given callback moment. Thus, `before_CONFIGURE+2` will complete execution strictly after `before_CONFIGURE` runs, but strictly before `enter_CONFIGURED-666` is executed.
-
 ### Call roles
 
 Call roles represent calls to integrated services. They must contain a `call`
@@ -212,7 +197,7 @@ for examples of call roles that reference a variety of integration plugins.
 
 The state machine callback moments are exposed to the AliECS workflow template interface and can be used as triggers or synchronization points for integration plugin function calls. The `call` block can be used for this purpose, with similar syntax to the `task` block used for controllable tasks. Its fields are as follows.
 * `func` - mandatory, it parses as an [`antonmedv/expr`](https://github.com/antonmedv/expr) expression that corresponds to a call to a function that belongs to an integration plugin object (e.g. `bookkeeping.StartOfRun()`, `dcs.EndOfRun()`, etc.).
-* `trigger` - mandatory, the expression at `func` will be executed once the state machine reaches this moment.
+* `trigger` - mandatory, the expression at `func` will be executed once the state machine reaches this moment. For possible values, see [State machine triggers](/docs/handbook/operation_order.md#state-machine-triggers)
 * `await` - optional, if absent it defaults to the same as `trigger`, the expression at `func` needs to finish by this moment, and the state machine will block until `func` completes.
 * `timeout` - optional, Go `time.Duration` expression, defaults to `30s`, the maximum time that `func` should take. The value is provided to the plugin via `varStack["__call_timeout"]` and the plugin should implement a timeout mechanism. The ECS will not abort the call upon reaching the timeout value!
 * `critical` - optional, it defaults to `true`, if `true` then a failure or timeout for `func` will send the environment state machine to `ERROR`.