ServiceComposer without HTTP context and ideas

[promontis]

Hi Mauro!

We've mailed before (a couple of years ago already... time moves fast). Recently, I've been involved with a new project that needs view model composition. Eventually we ended up writing our own implementation. However, as the project is ending, I still wanted to ask you about some of the ideas behind ServiceComposer so that I can use it in next projects instead of rolling our own.

ServiceComposer without HTTP context
Currently ServiceComposer relies on the HTTP context. However, I often find the need to also have a composition engine outside the HTTP context. For example, Search, Pricing, etc. I do see that you started with ServiceComposer.InMemory years ago. Was the idea to first create a ServiceComposer without HTTP context, and then build a ServiceComposer.AspNetCore on top of that? If so, that would be awesome!

In other words, I'd love to use ServiceComposer for other non-HTTP compositions. There is no HTTP request.

Would love to hear your view on this.

Strongly-typed models
ServiceComposer now supports using strongly typed view models. They have the advantages of strong typing and compiler checks and the disadvantages of a bit of coupling. Our team actually liked this as it was already using strongly typed models, coming from a more monolith codebase. Also for generating typescript types.

We currently put those ViewModel types in IT/Ops. It introduces a bit of coupling, but we are ok with that.

The implementation of IEndpointScopedViewModelFactory feels of little bit complicated. I've given you access to the private repo of how we implemented our composition engine. It is not as feature rich as this library, and I think it was version 1, but it shows you how we did it ourselves (I actually didn't code it, but I do like the general idea). It is here: https://github.com/promontis/composition-engine (you should have access now).

Notice the ICompositionRequestHandler<TRequest, TResponse>. So we basically type the Response, but also the Request (this is related to my first question... there is no http context here). Both types end up in IT/Ops. What do you think of this?

Traceability
In a previous project (years ago), I also rolled my own version based on the Workshop code. We also added support to see what data was appended by which appender/subscriber. Might be a nice feature to add.

In my current project we are also talking about generating some picture or doc-site where we show an overview of the routes and what appenders/subscribe are linked to them.

All in all, thanks for the great library! Hope to hear from you.

[mauroservienti]

Hi Mauro!

Good day, Michel.

We've mailed before (a couple of years ago already... time moves fast). Recently, I've been involved with a new project that needs view model composition. Eventually we ended up writing our own implementation. However, as the project is ending, I still wanted to ask you about some of the ideas behind ServiceComposer so that I can use it in next projects instead of rolling our own.

Glad to hear from you!

ServiceComposer without HTTP context Currently ServiceComposer relies on the HTTP context. However, I often find the need to also have a composition engine outside the HTTP context. For example, Search, Pricing, etc. I do see that you started with ServiceComposer.InMemory years ago. Was the idea to first create a ServiceComposer without HTTP context, and then build a ServiceComposer.AspNetCore on top of that? If so, that would be awesome!

In other words, I'd love to use ServiceComposer for other non-HTTP compositions. There is no HTTP request.

Would love to hear your view on this.

It's something I tried and then abandoned because I did not need it. It's still in my backlog of ideas to evolve the composition engine not to be so coupled to ASP.Net

Strongly-typed models ServiceComposer now supports using strongly typed view models. They have the advantages of strong typing and compiler checks and the disadvantages of a bit of coupling. Our team actually liked this as it was already using strongly typed models, coming from a more monolith codebase. Also for generating typescript types.

We currently put those ViewModel types in IT/Ops. It introduces a bit of coupling, but we are ok with that.

By view models here, do you mean the composition view model and the incoming view model that can be bound upon HTTP requests?

The implementation of IEndpointScopedViewModelFactory feels of little bit complicated. I've given you access to the private repo of how we implemented our composition engine. It is not as feature rich as this library, and I think it was version 1, but it shows you how we did it ourselves (I actually didn't code it, but I do like the general idea). It is here: https://github.com/promontis/composition-engine (you should have access now).

Yeah, I don't like those factories at all.
I'll have a look, thanks.

Notice the ICompositionRequestHandler<TRequest, TResponse>. So we basically type the Response, but also the Request (this is related to my first question... there is no http context here). Both types end up in IT/Ops. What do you think of this?

Yeah, I really wanted to get to something like what you have here with ICompositionRequestHandler<TRequest, TResponse>.
With what I'm working on right now in #804, the whole dependency on HTTP could probably be hidden or lifted more easily.

Traceability In a previous project (years ago), I also rolled my own version based on the Workshop code. We also added support to see what data was appended by which appender/subscriber. Might be a nice feature to add.

👍 I have OpenTelemetry support in the backlog — #685

In my current project we are also talking about generating some picture or doc-site where we show an overview of the routes and what appenders/subscribe are linked to them.

We started playing around with Swagger a while back (here is a repo with the spikes), but as usually happens time is limited and we abandoned it. It's still in the backlog, though.

All in all, thanks for the great library! Hope to hear from you.

Thank you!

[promontis]

By view models here, do you mean the composition view model and the incoming view model that can be bound upon HTTP requests?

Yeah both. So the Response (the viewmodel itself) and the Request itself. Them both being typed would be nice. We do see that it introduces coupling, but we don't think it's that bad; the flexibility and decoupling for us is mostly within the appenders and the subscribers (in-memory pub/sub).

Yeah, I really wanted to get to something like what you have here with ICompositionRequestHandler<TRequest, TResponse>.
With what I'm working on right now in #804, the whole dependency on HTTP could probably be hidden or lifted more easily.

I'll have a look at #804!

I also have a different question: did you ever think of adding viewmodel decomposition to this library? Or is it already supported?

[mauroservienti]

The response view model causes coupling, there could be mitigation techniques but I'm not sure they're worth it.

When it comes to requests binding each handler can have its private view model, and this is thanks to the fact that ServiceComposer allows multiple binding operations to a single incoming request.

Decomposition is already supported. You can have HttpPost or HttpPatch handlers, for example, and considering that each one can have a different binding model, they can extract the information they need.

I think this conversation is worth a blog post, thanks for raising the topic.

[markphillips100]

Hi @mauroservienti , been a long time since we discussed similar concepts, and after reading this issue/discussion I thought I'd give an update. My apologies for not continuing the discussion by the way. Time pressures meant going with a custom solution to implement the in-memory aspect @promontis mentions - had to refactor a fair bit to support the abstractions for non-http use cases. Maybe I'll push my implementation to a fork one day.

For in-memory I opted for abstracting ICompositionEndpoint<TRequest, TResult> and providing different base implementations to support the original HttpRequest mode and also in-memory (albeit I poorly named it "ObjectRequest"). This also requires an abstraction to composition context that ties the context to the types of endpoints.

NOTE: after reading your recent blogs it's likely that I've gone down an incompatible tangent with these abstractions, at least in terms of "feeling-at-home". I'd have to delve deeper to see if some alignment were possible.

Handlers of any endpoint type implementation are all scanned as usual and registered in a single registry, but they are filtered when requested for a specific route. So it's not just the http route that selects the handlers, but also the type of handler composition endpoint.

This of course means in-memory handlers, or any other variation of composition endpoint type, all still rely on http verbs for routes. This might seem counter to "in-memory" with it being nothing to do with http but it was a design choice so that route templating (for handler grouping) and model binding can still function the same. We need some way to couple endpoints that should be grouped together after all.

End result is I can implement the following. Note, I also added support for swagger api documentation but it is very opinionated based upon how I handle list composition. We discussed this before I think. It's hard.

A ViewCart handler for two different ICompositionContext<,>, one being a httprequest, and another to support in-memory:

    public class ViewCartGetHandler :
        ICompositionRequestsHandler<ICompositionContext<HttpRequest, IActionResult>>,
        ICompositionRequestsHandler<ICompositionContext<ObjectRequest, Result<DynamicViewModel>>>
    {
        private readonly IMediator _mediator;

        public ViewCartGetHandler(IMediator mediator)
        {
            _mediator = mediator;
        }

        [Tags("Customer Orders")]
        [ApiParameterDescription(Name = "orderId", IsRequired = true, Type = typeof(Guid), Source = "Path")]
        [ProducesResponseType(typeof(ProblemDetails), StatusCodes.Status400BadRequest)]
        [ProducesCompositionResponseType("sales", typeof(CartViewModel))]
        [Authorize(Policy = RootNamespacePrefixConstants.AuthenticationPolicy)]
        [HttpGet("api/sales/carts/{orderId:guid}")]
        public async Task Handle(ICompositionContext<HttpRequest, IActionResult> compositionContext)
        {
            var requestModel = await compositionContext.Request.Bind<ViewCartQuery.Query>();
            var result = await _mediator.Send(requestModel);

            var resultHandler = await result.HandleResultForCompositionRequestsAsync(compositionContext);

            await resultHandler
                .MatchResultForComposition(
                    async value => await ProcessValueAsync(compositionContext, requestModel, value)
                );
        }

        [HttpGet("/sales/carts/{orderId:guid}")]
        public async Task Handle(ICompositionContext<ObjectRequest, Result<DynamicViewModel>> compositionContext)
        {
            var requestModel = await compositionContext.Request.Bind<ViewCartQuery.Query>();
            var result = await _mediator.Send(requestModel);

            var resultHandler = await result.HandleResultForCompositionRequestsAsync(compositionContext);

            await resultHandler
                .MatchResultForComposition(
                    async value => await ProcessValueAsync(compositionContext, requestModel, value)
                );
        }

        private static async Task ProcessValueAsync<TRequest, TResult>(ICompositionContext<TRequest, TResult> compositionContext, ViewCartQuery.Query requestModel, CartViewModel value)
        {
            compositionContext.ViewModel.AppendValue(value);

            var orderItemIds = value.OrderItemDetails
                .SelectMany(orderItemDetail => orderItemDetail.OrderItem.GetAllOrderItems())
                .Select(orderItem => orderItem.Id)
                .Distinct()
                .ToArray();

            await compositionContext
                .RaiseEvent(new ViewCartRequested
                {
                    OrderId = requestModel.OrderId,
                    SellerId = value.OrderContext.SellerId,
                    SellerMenuConfigurationId = value.OrderContext.SellerMenuConfigurationId,
                    MenuId = value.OrderContext.MenuId,
                    OrderItemIds = orderItemIds
                });
        }
    }

    public static class OrderItemExtensions
    {
        public static IEnumerable<OrderItemViewModel> GetAllOrderItems(this OrderItemViewModel orderItem)
        {
            foreach (var item in orderItem.SelectedModifierGroups.SelectMany(modifierGroup => modifierGroup.SelectedItems.SelectMany(child => child.GetAllOrderItems())))
            {
                yield return item;
            }

            if (orderItem != null)
                yield return orderItem;
        }
    }

And a subscriber for both use cases:

    public class ViewCartGetSubscriber :
        ICompositionEventsSubscriber<ICompositionContext<HttpRequest, IActionResult>>,
        ICompositionEventsSubscriber<ICompositionContext<ObjectRequest, Result<DynamicViewModel>>>
    {
        private readonly IMediator _mediator;

        public ViewCartGetSubscriber(IMediator mediator)
        {
            _mediator = mediator;
        }

        [ProducesResponseType(typeof(ProblemDetails), StatusCodes.Status400BadRequest)]
        [ProducesCompositionResponseType(DynamicViewModelExtensions.ViewModelKey, typeof(CartViewModel))]
        [ApiParameterDescription(Name = "orderId", IsRequired = true, Type = typeof(Guid), Source = "Path")]
        [HttpGet("api/sales/carts/{orderId:guid}")]
        public void Subscribe(ICompositionEventsPublisher<ICompositionContext<HttpRequest, IActionResult>> publisher)
        {
            publisher.Subscribe<ViewCartRequested>(async (_, compositionContext) =>
            {
                var requestModel = await compositionContext.Request.Bind<ViewCartQuery.Query>();
                var result = await _mediator.Send(requestModel);

                var resultHandler = await result.HandleResultForCompositionRequestsAsync(compositionContext);

                await resultHandler
                    .MatchResultForComposition(
                        async value => await ProcessValueAsync(compositionContext, value)
                    );
            });
        }

        [HttpGet("/sales/carts/{orderId:guid}")]
        public void Subscribe(ICompositionEventsPublisher<ICompositionContext<ObjectRequest, Result<DynamicViewModel>>> publisher)
        {
            publisher.Subscribe<ViewCartRequested>(async (_, compositionContext) =>
            {
                var requestModel = await compositionContext.Request.Bind<ViewCartQuery.Query>();
                var result = await _mediator.Send(requestModel);

                var resultHandler = await result.HandleResultForCompositionRequestsAsync(compositionContext);

                await resultHandler
                    .MatchResultForComposition(
                        async value => await ProcessValueAsync(compositionContext, value)
                    );
            });
        }

        private static Task ProcessValueAsync<TRequest, TResult>(ICompositionContext<TRequest, TResult> compositionContext, CartViewModel value)
        {
            compositionContext.ViewModel.AppendValue(value);
            return Task.CompletedTask;
        }
    }

This then allows me to obtain a composition in-memory or (in-process as I like to refer to it):

    public sealed class ViewCartService : IViewCartService
    {
        private readonly ICompositionEndpoint<ObjectRequest, Result<DynamicViewModel>> _compositionEndpoint;
        private readonly ILogger _logger;

        public ViewCartService(
            ICompositionEndpoint<ObjectRequest, Result<DynamicViewModel>> compositionEndpoint,
            ILogger<ViewCartService> logger)
        {
            _compositionEndpoint = compositionEndpoint;
            _logger = logger;
        }


        public async Task<Result<DynamicViewModel>> GetCart(Guid orderId, IMessageHandlerContext context)
        {
            var queryString = context.MessageHeaders[PropagateOriginatingQueryStringHeaderBehavior.HeaderName];
            var request = new ObjectRequest(HttpMethods.Get, $"/sales/carts/{orderId}{queryString}");

            var result = await _compositionEndpoint.HandleAsync(request);
            if (result.IsFailed)
            {
                _logger.LogError("Error calling composition endpoint: {errors}", string.Join(", ", result.Errors.Select(x => x.Message).ToArray()));
            }

            return result;
        }

    }

From a use case standpoint, this allows me to propagate new data back to a client after an asynchronous process using signalr to push the data, rather than just publish a notification to tell the client to refresh and make a http call to a http-based composition for the same data.

This gets even more useful for use cases that might have many clients interested in the same data change. A notification to those clients to trigger api calls adds unnecessary load on the api. Using this in-memory approach where possible means the change is queried and propagated more optimally.