diff --git a/doc/specific_iocs/dae/Datastreaming.md b/doc/specific_iocs/Datastreaming.md
similarity index 66%
rename from doc/specific_iocs/dae/Datastreaming.md
rename to doc/specific_iocs/Datastreaming.md
index d02dcfcb0..baec4abdd 100644
--- a/doc/specific_iocs/dae/Datastreaming.md
+++ b/doc/specific_iocs/Datastreaming.md
@@ -9,7 +9,7 @@
datastreaming/*
```
-The data streaming system is being built as a requirement for HRPD-X and possibly SANDALS-II, separate (and complementary) to the `MNeuData` project. HRPD-X, SANDALS-II and other future instruments will not have a traditional DAE2/DAE3 as they are now obsolete. It is architecturally similar to the system that the ESS uses to take data (neutron events, sample environment, and anything else that we can throw into a streaming platform) and write it to file. Previously ISIS aided development to the ESS' streaming pipeline as part of an in-kind project. The system will replace the ICP at ISIS.
+The data streaming system is being built as a requirement for HRPD-X and possibly SANDALS-II, separate (and complementary) to the `MNeuData` project. HRPD-X, SANDALS-II and other future instruments will not have a traditional DAE2/DAE3 as they are now obsolete. It is architecturally similar to the system that the ESS uses to take data (neutron events, sample environment, and anything else that we can throw into a streaming platform) and write it to file. Previously ISIS aided development to the ESS' streaming pipeline as part of an in-kind project. The system will replace the ICP at ISIS. Note that all documentation here is applicable to the HRPD-X upgrade and may change in the future.
In general this works by producing both neutron events and histograms, sample environment data, and other diagnostic data into a [Kafka](https://kafka.apache.org/) cluster and having clients (consumers in Kafka lingo!) that either view data live and act on it or write the data to a nexus file. Additional information can be found [here](http://accelconf.web.cern.ch/AccelConf/icalepcs2017/papers/tupha029.pdf) and [here](https://iopscience.iop.org/article/10.1088/1742-6596/1021/1/012013).
@@ -17,12 +17,12 @@ All data is serialised into [Flatbuffers](https://flatbuffers.dev/) blobs using
Overall architecture is as follows:
-
+
This comprises of a few different consumers and producers:
-- [`azawakh`](https://github.com/ISISComputingGroup/azawakh) - This is a soft IOC which provides `areaDetector` views, spectra plots and so on by consuming events from the cluster and displaying them over EPICS CA/PVA.
-- [`borzoi`](https://github.com/ISISComputingGroup/borzoi) - This is also a soft IOC which is more or less a drop-in replacement for the ISISDAE. It provides an interface that several clients (ie. [genie](https://github.com/ISISComputingGroup/genie), [ibex_bluesky_core](https://github.com/ISISComputingGroup/ibex_bluesky_core), [ibex_gui](https://github.com/ISISComputingGroup/ibex_gui)) talk to to start/stop runs and configure streaming electronics. `borzoi` will send UDP packets to the streaming electronics to configure it.
-- [`BSTOKAFKA`](https://github.com/ISISComputingGroup/BSKAFKA) - This configures the `forwarder` with the blocks that are in an instrument's current configuration, as well as other PVs which will either get written to a file or archived for e.g. the log plotter.
+- [`kafka_dae_diagnostics`](https://github.com/ISISComputingGroup/kafka_dae_diagnostics) - This is a soft IOC which provides `areaDetector` views, spectra plots and so on by consuming events from the cluster and displaying them over EPICS CA/PVA.
+- [`kafka_dae_control`](https://github.com/ISISComputingGroup/kafka_dae_control) - This is also a soft IOC which is more or less a drop-in replacement for the ISISDAE. It provides an interface that several clients (ie. [genie](https://github.com/ISISComputingGroup/genie), [ibex_bluesky_core](https://github.com/ISISComputingGroup/ibex_bluesky_core), [ibex_gui](https://github.com/ISISComputingGroup/ibex_gui)) talk to to start/stop runs and configure streaming electronics. `kafka_dae_control` will send UDP packets to the streaming electronics to configure it.
+- [`kafka_forwarder_configurer`](https://github.com/ISISComputingGroup/kafka_forwarder_configurer) - This configures the `forwarder` with the blocks that are in an instrument's current configuration, as well as other PVs which will either get written to a file or archived for e.g. the log plotter.
- `forwarder` - See [Forwarding Sample Environment](datastreaming/Datastreaming---Sample-Environment)
- `filewriter` - See [File writing](datastreaming/Datastreaming---File-writing)
@@ -33,7 +33,7 @@ There is a (non-production!) [Redpanda](https://www.redpanda.com/) Kafka cluster
A web interface is available [here](https://reduce.isis.cclrc.ac.uk/redpanda-console/overview).
:::{important}
-It was decided that we no longer maintain the Kafka cluster, and it will be handled by the the Flexible Interactive
+It was decided that we no longer maintain the Kafka cluster, and it will be handled by the Flexible Interactive
Automation team. See `\\isis\shares\ISIS_Experiment_Controls\On Call\autoreduction_livedata_support.txt` for their
support information.
:::
diff --git a/doc/specific_iocs/dae/ISISDSLayout.drawio.svg b/doc/specific_iocs/dae/ISISDSLayout.drawio.svg
deleted file mode 100644
index edfcb8dfc..000000000
--- a/doc/specific_iocs/dae/ISISDSLayout.drawio.svg
+++ /dev/null
@@ -1,4 +0,0 @@
-
-
-
-
\ No newline at end of file
diff --git a/doc/specific_iocs/dae/datastreaming/Datastreaming--neutron-events-histograms.md b/doc/specific_iocs/dae/datastreaming/Datastreaming--neutron-events-histograms.md
deleted file mode 100644
index d92ea0e5b..000000000
--- a/doc/specific_iocs/dae/datastreaming/Datastreaming--neutron-events-histograms.md
+++ /dev/null
@@ -1,13 +0,0 @@
-{#dseventshistos}
-# Data streaming: Neutron events and histograms
-
-## For DAE2/DAE3 instruments
-The ICP (communicated to via the ISISDAE IOC) is responsible for communicating with the DAE2/DAE3 in terms of setting configuration, as well as streaming events and histograms from both.
-
-
-## For new instruments using FPGA-based acquisition electronics
-`borzoi` is responsible for communicating with the electronics and sending run starts/stops. It will have a similar interface to `ISISDAE` so we can drop-in replace it in the GUI.(?)
-
-
-## Live view, spectra plots etc.
-These will be provided by a soft IOC (`azawakh`) which effectively consumes from event and histogram topics (and possibly run starts?) which will serve areaDetector and other PVs.
diff --git a/doc/specific_iocs/dae/datastreaming/Datastreaming-run-starts-stops.md b/doc/specific_iocs/dae/datastreaming/Datastreaming-run-starts-stops.md
deleted file mode 100644
index 1baf8b892..000000000
--- a/doc/specific_iocs/dae/datastreaming/Datastreaming-run-starts-stops.md
+++ /dev/null
@@ -1,5 +0,0 @@
-{#dsrunstartstops}
-# Data streaming: run starts/stops
-
-Run starts and stops will be dealt with by [`borzoi`](https://github.com/ISISComputingGroup/borzoi) and the flatbuffers blobs will be constructed in this process. It may need to be hooked onto by `ISISDAE` for older instruments using DAE2/DAE3 and the ISISICP.
-
diff --git a/doc/specific_iocs/datastreaming/ADRs.md b/doc/specific_iocs/datastreaming/ADRs.md
new file mode 100644
index 000000000..a627aea0d
--- /dev/null
+++ b/doc/specific_iocs/datastreaming/ADRs.md
@@ -0,0 +1,9 @@
+# Data streaming: ADRs
+
+```{toctree}
+:glob:
+:titlesonly:
+:maxdepth: 1
+
+ADRs/*
+```
diff --git a/doc/specific_iocs/datastreaming/ADRs/000_kafka.md b/doc/specific_iocs/datastreaming/ADRs/000_kafka.md
new file mode 100644
index 000000000..aac8ddae5
--- /dev/null
+++ b/doc/specific_iocs/datastreaming/ADRs/000_kafka.md
@@ -0,0 +1,44 @@
+# 0 - Kafka
+
+## Status
+
+Accepted
+
+## Context
+
+We need to decide on a technology through which we are going to do data streaming.
+
+There are several options here:
+- Kafka or Kafka compatible solutions such as Redpanda
+- Redis
+- ZeroMQ/RabbitMQ/ActiveMQ
+
+Within each of these options we need to decide on a serialization format.
+Options are:
+- protobuffers
+- flatbuffers with ESS schemas
+- JSONB
+- msgpack
+- Avro
+- encoded JSON/BSON
+
+
+## Decision
+
+We have decided to use a Kafka compatible broker as a streaming platform. This may be either Kafka or Redpanda.
+
+This is because we can lean on the ESS experience in using this technology and may be able to collaborate with them and use shared tools.
+Flatbuffers encoding was performance tested during the in-kind project and showed good performance versus the alternatives at the time.
+
+We have also decided to serialize the data using the [ESS flatbuffers schemas](https://github.com/ess-dmsc/streaming-data-types) with ISIS additions where necessary.
+
+Kafka is a broker-based streaming technology - as opposed to brokerless systems which do not keep messages. This allows a Kafka-based system to replay messages or for a consumer to catch up with the 'history' of a stream. We will not retain events in Kafka indefinitely - retention will be tuned to keep a suitable number of messages for our use-cases versus hardware constraints.
+
+## Consequences
+
+What becomes easier or more difficult to do because of this change?
+
+Kafka is indisputably harder to set up than some other simpler alternatives. This is somewhat mitigated by its scaling and redundancy benefits.
+We don't intend to do a large amount in Kafka itself (ie. transforms or stream processors)
+
+The advantage of using Kafka is that we keep much more closely aligned to the ESS, CLF, ANSTO and other facilities who are all using Kafka with Flatbuffers schemas.
diff --git a/doc/specific_iocs/datastreaming/ADRs/001_histograms.md b/doc/specific_iocs/datastreaming/ADRs/001_histograms.md
new file mode 100644
index 000000000..08caccb08
--- /dev/null
+++ b/doc/specific_iocs/datastreaming/ADRs/001_histograms.md
@@ -0,0 +1,32 @@
+{#001_histograms}
+# 1 - Histograms and event mode
+
+## Status
+
+Pending discussion with HRPD-X interested parties (including instrument scientists & Mantid).
+
+## Context
+
+**Histogram mode**
+
+In histogram mode, over the course of a run, counts are accumulated into a running histogram, binned by user-specified
+time channel boundaries.
+
+**Event mode**
+
+In event mode, over the course of a run, each individual neutron event's detection time and detector ID is recorded.
+Event mode data can be later binned to form a histogram, but a histogram cannot be recovered to individual events. In
+other words, histogramming is lossy. The advantage of histogram mode is that it typically produces smaller data volumes.
+
+Histogram mode has historically been used due to hardware limitations in many cases.
+
+## Decision
+
+For HRPD-x, we will collect all data, including data from neutron monitors, in event mode only. HRPD-x will not support
+histogram mode.
+
+## Consequences
+
+- Data volumes on HRPD-x will be higher running in event mode compared to histogram mode. This includes both data in-flight
+during networking and Kafka processing, as well as final Nexus file sizes.
+- Only considering events will simplify components of the HRPD-x data streaming implementation.
diff --git a/doc/specific_iocs/datastreaming/ADRs/002_spectra_mapping.md b/doc/specific_iocs/datastreaming/ADRs/002_spectra_mapping.md
new file mode 100644
index 000000000..d8bd84ff6
--- /dev/null
+++ b/doc/specific_iocs/datastreaming/ADRs/002_spectra_mapping.md
@@ -0,0 +1,29 @@
+# 2 - Wiring and Spectra mapping
+
+## Status
+
+pending
+
+## Context
+
+Wiring tables is a concept that still exists however the format is now different (`.csv` as opposed to the old wiring table format).
+
+The options that we're considering are:
+- change `.csv` to align with the old format
+- write a service/script to convert to/from `.csv`
+- keep the two formats separate, acknowledging that they will not be backwards or forwards compatible
+
+Spectra files share the above considerations as they also use a different file format.
+
+Grouping spectra in hardware was primarily used to get around limitations of DAE hardware. In event mode there is no advantage to grouping spectra in hardware.
+
+## Decision
+
+We are not going to support the old-style spectra files or any spectrum mapping/grouping in general
+
+For wiring tables this is TBD in https://github.com/ISISComputingGroup/DataStreaming/issues/27.
+
+## Consequences
+
+- If HRPD-x previously grouped spectra in hardware, they will now need to be grouped in software (e.g. Mantid) instead.
+- Our data streaming software will not need to support spectrum grouping.
diff --git a/doc/specific_iocs/datastreaming/ADRs/003_linux.md b/doc/specific_iocs/datastreaming/ADRs/003_linux.md
new file mode 100644
index 000000000..077edcbf1
--- /dev/null
+++ b/doc/specific_iocs/datastreaming/ADRs/003_linux.md
@@ -0,0 +1,121 @@
+# 3 - Linux
+
+## Status
+
+Accepted.
+
+## Context
+
+Historically we have used Windows for running most of our software due to LabVIEW support when migrating from SECI.
+This often means software is not natively supported as most of the wider experiment controls community at comparable facilities run Linux instead, so we regularly have to port software designed for Linux to run on Windows.
+
+Streaming software is linux-centric and running Kafka itself natively on Windows is not natively supported by either [confluent Kafka](https://www.confluent.io/blog/set-up-and-run-kafka-on-windows-linux-wsl-2/#:~:text=Windows%20still%20isn%E2%80%99t%20the%20recommended%20platform%20for%20running%20Kafka%20with%20production%20workloads) or [redpanda](https://docs.redpanda.com/current/get-started/quick-start/#:~:text=If%20you%E2%80%99re,Linux%20%28WSL%29%2E). In addition, a number of tools developed both externally and at ISIS for data streaming only currently run on Linux.
+
+Running software under WSL is an option, but is not [built for production use.](https://learn.microsoft.com/en-us/windows/wsl/faq#can-i-use-wsl-for-production-scenarios-) It also adds several layers of complexity with networking, file systems and so on. Container networking configuration is difficult and limited using WSL.
+
+We would ideally like to run the data streaming software in containers as they offer security, deployment and repeatability benefits and are extremely popular in the software development industry. HRPD-X coming online with a new detector technology means that we will need to be able to easily apply new versions of data-streaming software at short notice. Containers give us an easy, repeatable path to be able to do this.
+
+Docker Desktop _is_ supported on Windows, but uses the WSL with a strict licensing agreement which does not suit our needs. Other alternatives also use the WSL. Many container configuration options (e.g. host networking, volume mounting options) cannot be supported with containers on Windows (whether via Docker desktop or another solution).
+
+One of the long-term goals on the Experiment Controls road-map is to revisit our operating system choice. It is very likely this will be a distribution of Linux.
+
+For HRPD-X specifically, the NDX and NDH is unsuitable for deploying data streaming software, as it will remain using Windows (in the short term).
+
+Additionally, Windows Licensing costs have recently changed significantly (as of January 2026). If we chose to use a Linux distribution, even with support ie. in the style of Red Hat Enterprise Linux, it is likely it would cost less.
+
+ISIS has made it clear that Linux support will be provided in the medium term.
+
+There are a few different options with associated risks and benefits:
+
+### 1 - Running natively on the (Windows) NDX machines
+
+This has the benefit that everything runs along with the rest of IBEX, though we would have to spend extra development effort trying to port software to Windows which we may undo in the future if moving to Linux anyway.
+
+The NDX already exists so doesn't add any system administration requirements to be able to run the software.
+
+The main drawbacks for this approach are that:
+- the NDXes do not have sufficient resources for the software currently
+- Windows is different to everyone else running the streaming software, and we may waste effort porting software to Windows
+- Kafka itself will not run on Windows
+- Deployment and patching will become difficult for a system which will need to be patched frequently and will interfere with the rest of the control system. As an example, we do not use virtual environments in python, so updating a python dependency for the `forwarder` runs the risk that it may break something in user scripts and/or the block server, which would be very bad.
+- Task scheduling/service management can be difficult on Windows, we could use `procserv` but in most cases we don't require an interactive terminal for processes.
+- Moving away from Windows is on the road-map - we would be creating more work for ourselves when we migrate over eventually.
+
+
+### 2 - Running natively on a separate Linux machine
+
+This adds another machine in the data streaming stack which _could_ fail and stop data collection.
+
+Running the streaming software (which is designed to be run on Linux) natively would be the most performant of all the approaches.
+
+The main benefit to this is that we can specify the hardware requirements independently so they are suitable for the streaming software, with enough overhead to run Kafka on if needed.
+
+Tooling is generally more available for deployment and patching than Windows, however this is not something we are familiar with as a team.
+
+The downside is that if the setup of this machine isn't entirely automated it could be very difficult to maintain and/or reproduce if a hardware failure occurred.
+
+As well as this, more Linux system administration knowledge is required by the team.
+
+Operating system updates could inadvertently affect the processes running on the machine, which could cause issues if we set the system to install unattended upgrades.
+
+### 3 - Running in containers on the NDX machines
+
+The benefit of doing this is that all services can be brought up and down together with the rest of the control system. It is also one less link in the data streaming chain to fail.
+
+This would require the use of the WSL which is not specifically designed for production use and has limited functionality with containers due to host-network mode issues and so on.
+NDXes are virtualised and have _very_ limited resources.
+
+Another drawback is that we currently the current IBEX deployment method makes it difficult to patch these services as easily - moving these services elsewhere means we do not have to interrupt e.g. sample environment scripts to restart/redeploy new versions of DAE processes.
+
+We are unable to use Docker desktop, required for docker engine on Windows, without paid licensing. An alternative is Rancher Desktop or podman. These both use underlying VMs.
+
+### 4 - Running in containers on a Linux VM on the NDH
+
+This has the same benefits as above but would allow us to run a Linux VM alongside the NDX which lets us avoid WSL oddities.
+Additionally, there should be more flexibility in deployment and patching, and less interference with the rest of IBEX.
+
+NDHes are currently very limited on resources, much like the NDXes which run on them. This is the main sticking point for this approach.
+
+Alongside this, virtual machines do generally introduce a performance penalty - the exact figure for this depends on several factors but it will never be as fast as a native application or a container which shares the kernel. For fast processes such as live histogramming and event processing we may require high performance which could be limited by a virtual machine.
+
+### 5 - Running in containers on a separate Linux machine
+
+This adds another machine in the data streaming stack which _could_ fail and stop data collection.
+As well as this, it will rely on the instrument's local network switch, which could also fail (though this is applicable to any of the options as the WLSF boards will be streaming over this switch)
+
+Another downside, which affects the above approach, is that some system administration knowledge will be required to keep the operating system alive and secure. If we are using containers this should be very minimal.
+
+This shares the benefit of being able to specify suitable hardware requirements as approach 2.
+
+Containers are much more easily reproducible than native software. The wider industry is moving towards them generally because of this amongst other reasons.
+
+Deployment, patching and orchestration is also very widely supported by several frameworks with containers. We should decide exactly what we're doing at a later point, but if we started with [`docker-compose`](https://docs.docker.com/compose/) as a simple first step, it is straightforward to move towards something like [Kubernetes](https://kubernetes.io/) instead if we decide we need the features it offers.
+
+Containers also provide a cyber-security benefit in that the processes are isolated individually.
+
+DSG already have some container-based software to convert UDP streams to Flatbuffers blobs - we could quite easily host this for them if we have container infrastructure. This applies to any of the approaches that offer it.
+
+If we end up being responsible for running the Kafka instance for HRPD-x, adding this is straightforward - Redpanda and the other Kafka implementations all offer production-ready container images.
+
+## Decision
+
+New hardware, running Linux, will be used to run the streaming software as shown {ref}`here`.
+
+Wherever possible, software will be deployed in containers, which will minimise the amount of Linux systems administration knowledge required. The aim will be for the Linux machine to 'only' have a container engine (such as docker or podman) installed, and very little else. Containers will use health checks and auto-restarting to ensure reliability. We will decide on exact deployment and orchestration methods later on, but there are several approaches to choose from.
+
+Exact specifications will depend on data rates and prototype testing.
+
+## Consequences
+
+- We are able to use Linux-centric technologies and tools, without needing to spend large amounts of time inventing workarounds for Windows.
+- The OS will be different. Developers will need _some_ understanding of Linux to maintain these servers.
+ * Mitigation: do as little as possible on the host, ideally limit it to just having a container engine installed via a configuration management tool such as Ansible. Some Linux distributions come with this out of the box such as [Fedora CoreOS](https://docs.fedoraproject.org/en-US/fedora-coreos/) or [RancherOS](https://rancher.com/docs/os/v1.x/en/)
+- Data-streaming infrastructure will not be on the NDH/NDX machine with the rest of IBEX. This is fine - EPICS is explicitly designed to run in a distributed way.
+- We will need to carefully consider the system specification of the Linux server in order to ensure it is adequate for expected data rates (including data rates from e.g. noisy detectors, to a point). In particular we expect to need to carefully consider:
+ * Disk write performance (for the Kafka broker and the filewriter)
+ * Network interface speeds (both from the electronics into this server, and from this server onwards to consumers such as Mantid)
+ * Memory (for any processes which need to histogram the data - must be able to keep a histogram in memory)
+ - The data streaming stack will be unaffected by a restart of the NDX system, and will keep running in the background.
+ - We will configure the relevant containers for data streaming software to automatically start on reboot of the data streaming Linux server.
+
+The above will be impacted if we are required to run a Kafka instance on the streaming machine - this is unclear as of January 2026. Redpanda provides [some documentation on hardware requirements](https://docs.redpanda.com/current/deploy/redpanda/manual/production/requirements/) which we should consider.
diff --git a/doc/specific_iocs/datastreaming/ADRs/004_vetos_runcontrol.md b/doc/specific_iocs/datastreaming/ADRs/004_vetos_runcontrol.md
new file mode 100644
index 000000000..4c343d6f8
--- /dev/null
+++ b/doc/specific_iocs/datastreaming/ADRs/004_vetos_runcontrol.md
@@ -0,0 +1,39 @@
+# 4 - Vetos and Run control
+
+## Status
+
+Accepted
+
+## Context
+
+**Vetos**
+Vetos are currently used by DAE2/DAE3 to mark neutron data as not useful - these are usually hardware signals fed into the DAE by other sources such as ISIS central timing systems and choppers.
+
+This will remain fairly similar for the streamed data and will be fed in via the VXI control boards, which has registers for viewing the status of vetos.
+
+There are two ways that an event packet could be vetoed, configurable via the VXI streaming control board (and the WLSF modules individually):
+- **Software veto:** Emit the frame, but mark it as invalid using veto flags
+- **Hardware veto:** Do not emit the frame
+
+Long term, we will likely need to allow for both modes. We _may_ also need to support each mode for each different type of veto - e.g. not emitting frames if one type of veto is active, while emitting frames marked as invalid for another type of veto.
+
+Data is still forwarded by UDP, but may not get processed into the `ev44` format if it is vetoed. This is configurable by the streaming boards.
+
+**Run control**
+
+Run control is controlled by IBEX using the existing {doc}`/system_components/Run-control` IOC. The concept of 'suspending' data collection if sample environment is outside a desired range will still be required.
+
+## Decision
+
+There will be a register, in the streaming control VXI crate that `kafka_dae_control` can write to, which will be set via EPICS by the run control IOC. This register will act exactly like a hardware veto signal, except will be controlled by software. The runcontrol status will be monitored by `kafka_dae_control`, and when it changes, `kafka_dae_control` will write to the corresponding register in the streaming control VXI crate.
+
+The overall concept of {external+ibex_user_manual:ref}`concept_good_raw_frames` will still be needed, as scientists will use {external+genie_python:py:obj}`genie.waitfor_frames` and similar functions to control their run durations.
+
+We have also agreed with DSG that the WLSF modules should _not_ be allowed to individually veto data despite this being technically possible and this should be the responsibility of the VXI control board.
+
+## Consequences
+
+- The existing concept of {doc}`/system_components/Run-control` is retained. This means that commands such as {external+genie_python:py:obj}`genie.waitfor_frames` will work largely as before, minimising required changes to instrument scripts.
+- The existing concepts of {external+ibex_user_manual:ref}`concept_good_raw_frames` are retained.
+- Existing vetoes will largely map across onto the new system.
+- `kafka_dae_control` will need to monitor the run control [this PV](https://github.com/ISISComputingGroup/EPICS-RunControl/blob/master/RunControlApp/Db/gencontrolMgr.db#L54C28-L54C35) for changes.
diff --git a/doc/specific_iocs/dae/datastreaming/Datastreaming---File-writing.md b/doc/specific_iocs/datastreaming/Datastreaming---File-writing.md
similarity index 96%
rename from doc/specific_iocs/dae/datastreaming/Datastreaming---File-writing.md
rename to doc/specific_iocs/datastreaming/Datastreaming---File-writing.md
index 9809278b4..05bd0572c 100644
--- a/doc/specific_iocs/dae/datastreaming/Datastreaming---File-writing.md
+++ b/doc/specific_iocs/datastreaming/Datastreaming---File-writing.md
@@ -1,4 +1,4 @@
-# File writing
+# Data streaming: File writing
The [filewriter](https://github.com/ess-dmsc/kafka-to-nexus) is responsible for taking the neutron and SE data out of Kafka and writing it to a nexus file. When the ICP ends a run it sends a config message to the filewriter, via Kafka, to tell it to start writing to file.
diff --git a/doc/specific_iocs/dae/datastreaming/Datastreaming---Sample-Environment.md b/doc/specific_iocs/datastreaming/Datastreaming---Sample-Environment.md
similarity index 95%
rename from doc/specific_iocs/dae/datastreaming/Datastreaming---Sample-Environment.md
rename to doc/specific_iocs/datastreaming/Datastreaming---Sample-Environment.md
index 4b17a6717..213cc3d1b 100644
--- a/doc/specific_iocs/dae/datastreaming/Datastreaming---Sample-Environment.md
+++ b/doc/specific_iocs/datastreaming/Datastreaming---Sample-Environment.md
@@ -1,4 +1,4 @@
-# Sample environment forwarding
+# Data streaming: Sample environment forwarding
All IBEX instruments are currently forwarding their sample environment PVs into Kafka. This is done in two parts:
diff --git a/doc/specific_iocs/datastreaming/Datastreaming--neutron-events-histograms.md b/doc/specific_iocs/datastreaming/Datastreaming--neutron-events-histograms.md
new file mode 100644
index 000000000..1342acdb4
--- /dev/null
+++ b/doc/specific_iocs/datastreaming/Datastreaming--neutron-events-histograms.md
@@ -0,0 +1,13 @@
+{#dseventshistos}
+# Data streaming: Neutron events and histograms
+
+## For DAE2/DAE3 instruments
+The ICP (communicated to via the ISISDAE IOC) is responsible for communicating with the DAE2/DAE3 in terms of setting configuration, as well as streaming events and histograms from both.
+
+
+## For new instruments using FPGA-based acquisition electronics
+`kafka_dae_control` is responsible for communicating with the electronics and sending run starts/stops. It will have a similar interface to `ISISDAE` so we can drop-in replace it in the GUI.(?)
+
+
+## Live view, spectra plots etc.
+These will be provided by a soft IOC (`kafka_dae_diagnostics`) which effectively consumes from event and histogram topics (and possibly run starts?) which will serve areaDetector and other PVs.
diff --git a/doc/specific_iocs/dae/datastreaming/Datastreaming-How-To.md b/doc/specific_iocs/datastreaming/Datastreaming-How-To.md
similarity index 98%
rename from doc/specific_iocs/dae/datastreaming/Datastreaming-How-To.md
rename to doc/specific_iocs/datastreaming/Datastreaming-How-To.md
index 50d69f501..aca53d288 100644
--- a/doc/specific_iocs/dae/datastreaming/Datastreaming-How-To.md
+++ b/doc/specific_iocs/datastreaming/Datastreaming-How-To.md
@@ -1,5 +1,5 @@
{#datastreaminghowto}
-# Data streaming how-to guide
+# Data streaming: how-to guide
This is a guide for basic operations using either the development or production Kafka clusters we use for data streaming at ISIS.
diff --git a/doc/specific_iocs/dae/datastreaming/Datastreaming-Topics.md b/doc/specific_iocs/datastreaming/Datastreaming-Topics.md
similarity index 90%
rename from doc/specific_iocs/dae/datastreaming/Datastreaming-Topics.md
rename to doc/specific_iocs/datastreaming/Datastreaming-Topics.md
index 6baa51204..1ed5f7acd 100644
--- a/doc/specific_iocs/dae/datastreaming/Datastreaming-Topics.md
+++ b/doc/specific_iocs/datastreaming/Datastreaming-Topics.md
@@ -1,9 +1,13 @@
-# Data streaming topics
+# Data streaming: topics
We have a number of topics per-instrument on `livedata`, the {ref}`Kafka cluster` we use.
+Partition numbers are listed below. For variable partitions this will depend on the throughput requirements of the specific instrument.
+
## `_runInfo`
+partitions: 1
+
This contains run start and run stop flatbuffers blobs.
Flatbuffers schemas in this topic:
@@ -12,6 +16,8 @@ Flatbuffers schemas in this topic:
## `_events`
+partitions: variable
+
This contains data from event-mode events.
Flatbuffers schemas in this topic:
@@ -19,6 +25,8 @@ Flatbuffers schemas in this topic:
{#topics_sampleenv}
## `_sampleEnv`
+
+partitions: 1
This contains sample environment data forwarded from EPICS.
In a `.nxs` file this should end up in `raw_data_1/selog/`
@@ -32,6 +40,8 @@ Flatbuffers schemas in this topic:
## `_runLog`
+partitions: 1
+
This contains run metadata forwarded from the ICP.
In a `.nxs` file this should end up in `raw_data_1/runlog/`
@@ -39,34 +49,46 @@ Schemas in this topic match the ones in {ref}`topics_sampleenv`
## `_monitorHistograms`
+partitions: variable
+
This contains monitor histograms.
Flatbuffers schemas in this topic:
- [`hs01` - Histograms](https://github.com/ess-dmsc/streaming-data-types/blob/master/schemas/hs01_event_histogram.fbs)
## `_detSpecMap`
+partitions: 1
+
This contains details of the detector-spectrum mapping.
Flatbuffers schemas in this topic:
- [`df12` - Detector-spectrum mapping](https://github.com/ess-dmsc/streaming-data-types/blob/master/schemas/df12_det_spec_map.fbs)
## `_areaDetector`
+partitions: variable
+
This is raw `areaDetector` data. It's sent by [this line in `ISISDAE`](https://github.com/ISISComputingGroup/EPICS-ioc/blob/716aada58c972cf0661ab6cebc41fba34d29b806/ISISDAE/iocBoot/iocISISDAE-IOC-01/liveview.cmd#L8)
## `_forwarderConfig`
+partitions: 1
+
This is the forwarder configuration, sent by {ref}`bskafka`.
Flatbuffers schemas in this topic:
- [`fc00` - Forwarder configuration](https://github.com/ess-dmsc/streaming-data-types/blob/master/schemas/fc00_forwarder_config.fbs)
## `_forwarderStatus`
+partitions: 1
+
This is the forwarder status topic which contains details about what PVs the forwarder is forwarding.
Flatbuffers schemas in this topic:
- [`x5f2` - General status](https://github.com/ess-dmsc/streaming-data-types/blob/master/schemas/x5f2_status.fbs)
## `_forwarderStorage`
+partitions: 1
+
This is the last known forwarder configuration, sent by {ref}`bskafka`. This is for if the forwarder crashes, then it can quickly retrieve its last configuration.
Flatbuffers schemas in this topic:
- [`fc00` - Forwarder Configuration](https://github.com/ess-dmsc/streaming-data-types/blob/master/schemas/fc00_forwarder_config.fbs)
diff --git a/doc/specific_iocs/datastreaming/Datastreaming-hardware-architecture.md b/doc/specific_iocs/datastreaming/Datastreaming-hardware-architecture.md
new file mode 100644
index 000000000..9f068ec05
--- /dev/null
+++ b/doc/specific_iocs/datastreaming/Datastreaming-hardware-architecture.md
@@ -0,0 +1,66 @@
+{#ds_hardware_architecture}
+
+# Data streaming: hardware architecture
+
+```{mermaid}
+architecture-beta
+ group control_board[CONTROL BOARD]
+ service timing_fanout(server)[Timing fanout board] in control_board
+ service vxi_control_board(server)[VXI Control board] in control_board
+
+ group wlsf_module[WLSF MODULE]
+ service detector_fpga(server)[WLSF Detector FPGA] in wlsf_module
+
+ group linux(server)[LINUX STREAMING SERVER]
+ service kafka_runInfo(database)[kafka runInfo] in linux
+ service kafka_udp(database)[kafka UDP] in linux
+ service udp(server)[udp to kafka Rust] in linux
+ service event_processor(server)[event processor Rust] in linux
+ service kafka_events(database)[kafka Events] in linux
+
+ group external_signals[EXTERNAL SIGNALS]
+ service isis_timing(server)[ISIS Timing TOF GPS PPP Vetos] in external_signals
+
+ group ibex[IBEX IOCs] in linux
+ service kafka_dae_control(server)[KDAE Control] in ibex
+ service kafka_dae_diagnostics(server)[KDAE Diagnostics] in ibex
+
+ udp:T --> B:kafka_udp
+ kafka_udp:R --> L:event_processor
+ event_processor:B --> T:kafka_events
+
+ detector_fpga:R --> L:udp
+ timing_fanout:T --> B:detector_fpga
+ vxi_control_board:T --> B:timing_fanout
+
+ vxi_control_board:B <-- T:isis_timing
+
+ kafka_dae_control:L --> R:vxi_control_board
+ kafka_events:B --> T:kafka_dae_diagnostics
+ kafka_dae_control:B --> T:kafka_runInfo
+```
+
+## Hardware components
+
+### VXI Control Board
+
+Each instrument will have exactly one VXI streaming control board. It controls "global" instrument state, gets timing signals and PPP fed into it. It it what `kafka_dae_control` primarily talks to begin and end event streaming.
+
+### Timing Fanout board
+
+This sends data from the VXI board to the (potentially multiple) WLSF modules. IBEX will not need to communicate to this.
+
+### WLSF Fibre module (FPGA)
+
+An instrument may have several of these, depending on the instrument detector configuration. These have a similar UDP-based interface to the VXI boards but also have settings for each detector module.
+
+IBEX will not need to talk to these except for "advanced" diagnostics.
+
+### Linux streaming server
+This hosts a Kafka cluster, for this bit of infrastructure there are three topics:
+- `runInfo` - this contains `pl72` and `6s4t` run starts/stops send by `kafka_dae_control`
+- `raw_udp` - this contains Kafka messages corresponding to each UDP packet which was received (with metadata such as IP address) sent by the [rust udp to Kafka process](https://gitlab.stfc.ac.uk/isis-detector-systems-group/software/data-streaming/rust-udp-to-kafka) which is also hosted on this server
+- `events` - this contains `ev44` formed by the [event stream processor](https://gitlab.stfc.ac.uk/isis-detector-systems-group/software/data-streaming/rust-data-stream-processor/-/tree/main?ref_type=heads) which is also hosted on this server
+
+
+This also hosts the `kafka_dae_control` and `kafka_dae_diagnostics` IOCs.
diff --git a/doc/specific_iocs/datastreaming/Datastreaming-run-starts-stops.md b/doc/specific_iocs/datastreaming/Datastreaming-run-starts-stops.md
new file mode 100644
index 000000000..4fd0b1bee
--- /dev/null
+++ b/doc/specific_iocs/datastreaming/Datastreaming-run-starts-stops.md
@@ -0,0 +1,9 @@
+{#dsrunstartstops}
+# Data streaming: run starts/stops
+
+Run starts and stops will be dealt with by [`kafka_dae_control`](https://github.com/ISISComputingGroup/kafka_dae_control) and the flatbuffers blobs will be constructed in this process. It may need to be hooked onto by `ISISDAE` for older instruments using DAE2/DAE3 and the ISISICP.
+
+Run starts will contain static and streamed data in the `nexus_structure`, including things like `run_number`, `instrument_name` and so on which will get written to a file.
+
+Run starts will _also_ contain metadata used by `kafka_dae_diagnostics`, in a json schema defined by https://github.com/ISISComputingGroup/DataStreaming/issues/29 - this is so `kafka_dae_diagnostics` does not have to try and parse the `nexus_structure` as it doesn't know or care about NeXus files.
+
diff --git a/doc/specific_iocs/datastreaming/ISISDSLayout.drawio.svg b/doc/specific_iocs/datastreaming/ISISDSLayout.drawio.svg
new file mode 100644
index 000000000..a3cbf29a4
--- /dev/null
+++ b/doc/specific_iocs/datastreaming/ISISDSLayout.drawio.svg
@@ -0,0 +1,4 @@
+
+
+
+
\ No newline at end of file
diff --git a/doc/specific_iocs/dae/ISISDSLayout.xml b/doc/specific_iocs/datastreaming/ISISDSLayout.xml
similarity index 97%
rename from doc/specific_iocs/dae/ISISDSLayout.xml
rename to doc/specific_iocs/datastreaming/ISISDSLayout.xml
index 78d0f1048..0bb1ceff8 100644
--- a/doc/specific_iocs/dae/ISISDSLayout.xml
+++ b/doc/specific_iocs/datastreaming/ISISDSLayout.xml
@@ -122,7 +122,7 @@
-
+
@@ -195,7 +195,7 @@
-
+
diff --git a/doc/spelling_wordlist.txt b/doc/spelling_wordlist.txt
index 385a44ba0..d69c71869 100644
--- a/doc/spelling_wordlist.txt
+++ b/doc/spelling_wordlist.txt
@@ -58,6 +58,7 @@ autosaved
autosaving
autostart
autotune
+Avro
backend
Baldor
Baratron
@@ -89,6 +90,7 @@ booleans
bootable
bottlenecking
breakpoint
+brokerless
bruteforce
bumpstrip
burndown
@@ -161,6 +163,7 @@ Culham
customisable
cxx
Cybaman
+cyber
cybersecurity
cygwin
DAC
@@ -420,6 +423,7 @@ junit
jvisualvm
Jython
Jülich
+Kafka
Kammrath
Kanban
kbaud
@@ -462,6 +466,7 @@ longin
longout
lookups
loopback
+lossy
lowlimit
lowT
LSi
@@ -531,6 +536,7 @@ moxas
MPa
msc
msg
+msgpack
msi
MSLITS
msm
@@ -637,6 +643,7 @@ PLCs
plt
PNG
png
+podman
polarisers
polref
postfixed
@@ -665,6 +672,7 @@ procserver
profiler
programmatically
proto
+protobuffers
pseudocode
psu
pugixml
@@ -879,6 +887,7 @@ ua
uA
uamps
UC
+UDP
UI
ui
uk
@@ -925,6 +934,7 @@ vhds
vhdx
viewmodel
virtualbox
+virtualised
vis
Viscotherm
vm