Emissions CSV export

[aaccensi]

Context

As part of the emissions project we want to be able to export the emissions data in a CSV. The existing ConfiguredCSVSerializer serializes graph data to CSV based on a YAML config in ETSource, but it only supported simple query-based columns. The emissions export requires iterating over nodes in a group and reading attributes directly from each node, including support for both the energy and molecules graphs.

Implemented changes

Expanded ConfiguredCSVSerializer to support two new column types:

• node_group: A hidden column whose value names a node group. The config row is expanded into one CSV row per node in that group, combining nodes from both the energy and molecules graphs.
• node_attribute: Evaluates a Ruby expression against node_api for each expanded node. Supports an optional transform: expression (with value bound to the result) for post-processing, e.g. unit conversion or conditional mapping.

Other changes:

• Added an optional period: parameter (:present or :future) to the serializer to select which graph to use for node expansion. Existing column types (present, future, unit, query) are unaffected.
• Fixed label: being silently ignored for non-query column types.
• Added emissions_present and emissions_future CSV endpoints backed by a new emissions_csv.yml config in ETSource.

Related

This effort is in 3 places but only 2 of them have PRs (to merge back into the general emissions branch)

• etengine: Emissions CSV export etengine#1753
• etsource: Emissions CSV export #3443
• etmodel: No PR, created emissions branch that didn't exist yet

Checklist

• I have tested these changes
• I have updated documentation as needed
• I have tagged the relevant people for review

[mabijkerk]

This seems a bit 'devvy' to be usable on the long term for modellers. Are there alternatives?

[aaccensi]

It is quite 'devy', @noracato loved it.
If we do not want any code in ETSource then we will implement individual methods for each new special column that needs a calculation or evaluation or odd way to access it. But if the argument is not to require a dev I disagree because you will need to involve a dev to create that exclusive method in the csv exporter anyway.

[kndehaan]

@aaccensi what would be a potential solution to not have this dev-y stuff in the yml, but in the engine method instead? How could the value: then be specified differently?

[noracato]

I think we can do a compromise in this case:

# node_api/emissions.rb

def ghg_carrier_key
  inputs.map { |s| s.carrier&.key }.include?(:other_ghg) ? :other_ghg : :co2
end

Like this it becomes a simple node attribute, scoped to the emissions module. Then in ETSource we can do like this:

- name: GHG
  type: node_attribute
  value: ghg_carrier_key

[aaccensi]

There is no complication whatsoever to move this to the parser in the engine, we can simply replace value: and transform: configs by anything sufficiently adequate and explanatory like subtype: ghg that would only do this specific thing we ask it to do.

Perhaps my point did not came across, my point is only about being versatile and create a generic solution that is easily expandable following the very engineering principles we recently defined.

[noracato]

I agree not to move specific methods in the serialiser as per the engineering principles. How do you feel about my solution to create a scoped method in the NodeApi that can be directly called by the node_attribute type in the serialiser?

[kndehaan]

I discussed with @noracato the potential options. Preference goes to reducing the complexity in the yml file by moving the method to the NodeApi. @aaccensi would you agree with this solution as well?

[mabijkerk]

I also feel that we typically solve this type of conversion in a different way, not with an explicit 1e-6 in ETSource. Are there alternatives?

[aaccensi]

This is an example of reusing the 'devy' method that you mentioned above, you can use it to do conversion, negations, any kind of mathematical mutation, or print labels based on conditional values, or many other things, it is very versatile by knowing a bit of ruby. Even if we add the dedicated method for the GHG column I think the "transform" method should stay.

[noracato]

also feel that we typically solve this type of conversion in a different way

Number conversions you refer to actually happen in ETModel. The emissions methods themselves just return a number float, they have no clue what kind of unit they are returning. So there is no easy way of automatic scaling possible with these results

[kndehaan]

Discussed with @noracato that there doesn't seem to be a better way other than putting it like this in the yml. Therefore, I think it's fine to leave it in here like it now is.

[mabijkerk]

The data export works and the config looks clear enough: the rows are not too much. The data export does generate various numbers of decimals. Perhaps it is good to determine a fixed amount of decimals?

[kndehaan]

The data export does generate various numbers of decimals. Perhaps it is good to determine a fixed amount of decimals?

Discussed with @noracato that we have two approaches to do this:

1. Limit the number of decimals in the transform function in the yml file: transform: "(value * 1e-6).round(3)". This does complicates it a bit more in the yml file however.
2. Apply the same decimals rounding to all data exports that are generated with the serializer and yml files. Then the rounding can be integrated in the serializer.

This is something that we should discuss further.