Feat: Build a new Graph as a combination of Graphs

[jbdebard]

Is your feature request related to a problem? Please describe.
I am creating specific Graphs at runtime as the combination of several Graphs depending on a configuration.

Describe the solution you'd like
I would need to define a Graph as a merge of 2 or more Graphs with additional transitions, or to be able to get both actions and transitions in the Graph API so that I can extract them from my "sub" graphs and can re-combine them while building a new Graph.

graph_1 = (
    GraphBuilder()
    .with_actions(**actions_1)
    .with_transitions(*transitions_1)
    .build()
)

graph_2 = (
    GraphBuilder()
    .with_actions(**actions_2)
    .with_transitions(*transitions_2)
    .build()
)

final_graph = (
    GraphBuilder()
    .with_graphs(graph_1, graph_2)
    .with_transitions(*additional_transitions)
    .build()

Describe alternatives you've considered
I could consider passing my 'sub' graphs as a set of actions and transitions instead of Graph, but I find this solution not very elegant and will add complexity to my code as I also might need to use the 'sub' graphs in standalone burr Applications.

Thanks for your feedback,
KR
jbd

[skrawcz]

@jbdebard thanks for the issue. I need to double check the code path whether we handle this in the API style you want, but we have a RunnableGraph concept that we created for parallelism that is conceptually what you describe. One wrinkle is that to link graphs we need to specify the start, end, and possibly some state stuff -- did you have any thoughts there?

How much of a blocker is this? Right now the way around it is (which is what I think you're referencing as what you're currently doing?) that you stick those two graphs into two actions and wire it like normal, with the actions then creating the right burr subgraph to invoke - see this part of the docs.

E.g. I think this might be a reasonable use case of the class based action approach

graph_1 = (
    GraphBuilder()
    .with_actions(**actions_1)
    .with_transitions(*transitions_1)
    .build()
)

graph_2 = (
    GraphBuilder()
    .with_actions(**actions_2)
    .with_transitions(*transitions_2)
    .build()
)


class SubGraphAction(Action):
    
    def __init__(self, sub_graph, reads, writes):
       # constructor
       self.sub_graph = sub_graph
       ...

    @property
    def reads(self) -> list[str]:
        return self.reads

    def run(self, state: State) -> dict:
        # create application given subgraph
        app = ApplicationBuilder().with_graph(self.sub_graph)... # etc
        stuff = app.run(...)
        return {"some_result": "to_save_to_state", ...}

    @property
    def writes(self) -> list[str]:
        return self.writes

    def update(self, result: dict, state: State) -> State:
        return state.update(**result) 


final_graph = (
    GraphBuilder()
    .with_graphs(SubGraphAction(graph1, ...).with_name("graph1"), SubGraphAction(graph_2, ...).with_name("graph2"))
    .with_transitions(*additional_transitions)
    .build()

Note: I haven't tested the above code ^ . @elijahbenizzy any thoughts on either approach?

[elijahbenizzy]

So yeah, it depends what you want to do. If its just the union of the two graphs, then we should be able to pretty easily allow you to add all the nodes from a graph... But that's not supported. Would be curious to get more of a sense of your use-case if you don't mind sharing?

Otherwise if you want to encapsulate them within an action what @skrawcz recommended is the right approach (and we can always add tooling to make that easier -- it is currently tailored towards parallelism through recursion, but it shouldn't necessarily be coupled to that...).

I think both are feasible, but my guess is you want the union approach from your wording.

Currently, we have with_graph which just adds a prebuilt graph and doesn't work with actions/transitions, but I don't see any reason it couldn't be additive. E.G. we couldn't do what you said, or just have multiple with_graph calls. Might have some time to hack it out to ensure that it's feasible -- will report back. Otherwise you can probably implement what you want with a bit of hacking -- have not tested this out (some nuances of how we do actions/assigning names could break it, but in theory it should work...).

Note this assumes that action names are not shared between actions.

Let us know what the preferred approach is and we're happy to scope it out/see if we can hack it in.

graph_1 = (
    GraphBuilder()
    .with_actions(**actions_1)
    .with_transitions(*transitions_1)
    .build()
)

graph_2 = (
    GraphBuilder()
    .with_actions(**actions_2)
    .with_transitions(*transitions_2)
    .build()
)

final_graph = (
    GraphBuilder()
    .with_actions(*[action for action in graph_1.actions + graph_2.actions]) # must be unique
    .with_transitions(*[(transition.from_.name, transition.to.name, transition.condition) for transition in graph_1.transitions + graph_2.transitions])
    .with_transitions(*additional_transitions)
    .build()

[elijahbenizzy]

OK had some fun with this -- mind trying out pip install burr==0.40.0rc0? Has the exact API you suggested.

See docs here

Additionally I think we'll want to enable @skrawcz's API> this is a limitation of the parallelism design that is coupled to recursion -- shouldn't be too tough to tease apart.

[skrawcz]

@jbdebard ^ see the RC version.

[jbdebard]

Hello @skrawcz and @elijahbenizzy. Thank you very much, I tested burr==0.40.0rc0 and it works just as expected for me (I used a workaround so far, but will refactor asap).

As for my use case actually, I'm building a pipeline factory that plugs different kinds of models with different kinds of preprocessing at runtime depending on an REST API request. The preprocessing tasks and the models are both burr.core.Graph (sometimes just a wrapper around a function, but some other times really needing Graph structure).

The preprocessing part should be pure DAG and I think maybe I should have used Hamilton instead, but I had only 2 month to get it in prod and no time to learn both libraries.

Anyway, a big thank to both of you for being so reactive 👌