# Lose provenance inside WorkChain **URL:** \ **Category:** General Usage\ **Created:** [October 13, 2023, 8:58am UTC](https://aiida.discourse.group/t/lose-provenance-inside-workchain/126 "2023-10-13T08:58:35Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Xing](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/xing/32/10_2.png) [@Xing](https://aiida.discourse.group/u/Xing)\ **Post date:** [October 13, 2023, 8:58am UTC](https://aiida.discourse.group/t/lose-provenance-inside-workchain/126/1 "2023-10-13T08:58:35Z")
When writing a workchain, especially when using the context, people can easily create data inside workchain and lose the provenance. For example, in the `aiida-quantumespresso`, the `PdosWorkChain` and `pwBandsWorkChain` have [lines of code](https://github.com/aiidateam/aiida-quantumespresso/blob/fc1a940d4a60f22b42ec0a069a06436c0c9ae0f5/src/aiida_quantumespresso/workflows/pw/bands.py#L234-L236) to generate new input parameters by using the `number of bands` from the previous `relax` calculation. In these cases, without looking at the source code, we will not know where these input parameters come from, and the logic of provenance is lost. In AiiDA doc, there is an [example](https://aiida.readthedocs.io/projects/aiida-core/en/latest/topics/workflows/concepts.html#implementation) to tell people not to create results (output) inside workchain. Thus, we also need clear documentation on not creating data (input) inside the workchain. ---
**Author:** ![danielhollas](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/danielhollas/32/4_2.png) [@danielhollas](https://aiida.discourse.group/u/danielhollas)\ **Post date:** [October 13, 2023, 10:23am UTC](https://aiida.discourse.group/t/lose-provenance-inside-workchain/126/2 "2023-10-13T10:23:15Z")
I have to say I have often been struggling with this, it can easily become very tedious especially if you need to make minor modifications in the input files (e.g. adding a WF guess from previous calc). So I am very interested what people think. ---
**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\ **Post date:** [October 13, 2023, 12:55pm UTC](https://aiida.discourse.group/t/lose-provenance-inside-workchain/126/3 "2023-10-13T12:55:14Z")
I had also pondered this question when I just joined the AiiDA team. At the time, the very first workflows (this was in the pre v1.0 era with a completely different system) would have a lot of `calcfunction`s to mutate `Dict` nodes to update workflow input parameters. This was a nightmare for maintenance, because every specific type of change required its own `calcfunction` implementation so we ended up with a great number of them. I tried to generalize some of them by having a generic `Dict` update `calcfunction` but this soon also became too complex or incomplete for various use cases such as nested dictionaries. A provenance-purist mindset had taken over and was trying to plug all holes where provenance was _leaking_. At that point, I took a step back and asked myself the question: _what is the real reason we are doing this?_ The point of tracking provenance is to enable retracing the origins of data generated by certain processes. Here we are mostly interested in _outputs_ of processes. Of course outputs can be become _inputs_ to other processes, so in that sense tracking inputs to processes are also important. However, some inputs do not have any real history, they are just “created out of thin air” at the start of a chain of processes. What is the real value that is created by tracking the modification of input parameters in nested workflows? Is any information really lost by not explicitly tracking the modification of input parameter dictionaries? At this point, a historical note is important. In the very first workflow system of AiiDA (the one I described above) the workflows did not live in the same graph as the calculations (as they do now). In fact, the workflows were not part of the provenance graph whatsoever. This means that the _only_ provenance that was tracked was that of calculations and their input and output data. This motivated the original workflow authors wanting to track input parameter mutations explicitly. With the current provenance paradigm of AiiDA, where workflows are fully part of the provenance graph, I would argue that information is not really even lost. Although the modification of the parameters may not be explicitly represented through a `calcfunction`, the _workflow_ records the context in which the modification took place. Since the version of the package, from which the workflow comes is recorded, which indirectly captures the source code, the logic of the modification is (albeit indirectly) also recorded. When we look just on the level of calculation provenance, the input may appear to simply pop into existence, but the same would go for input parameters for the top-level workflow. In the end, I approach this problem pragmatically: what is the problem that we are trying to solve? Is the explicit capturing of input parameter modification through a `calcfunction` really giving us valuable information that would otherwise be missing? My answer is _no_. We are currently not losing any valuable information. And the alternative is actually imposing a significant burden. It is making the code more complex, more difficult to maintain, and, arguably, the provenance graph is unnecessarily made more complex. It is currently not possible to capture perfect provenance and it might not even be necessary or useful. To me, AiiDA should provide users with the tools to track provenance as easily as possible and as complete as necessary. We should simply make it as clear as possible to users _when_ they run the risk of losing provenance and the implications. It is then up to them to decide what level of provenance is important to their use case. ---
**Author:** ![danielhollas](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/danielhollas/32/4_2.png) [@danielhollas](https://aiida.discourse.group/u/danielhollas)\ **Post date:** [October 14, 2023, 1:13pm UTC](https://aiida.discourse.group/t/lose-provenance-inside-workchain/126/4 "2023-10-14T13:13:20Z")
@sphuber this is gold, thanks so much for writing it up! : 👏 I definitely agree with the general sentiment. I wonder if some of this wisdom should be captured in the docs (if it isn’t already). ---
**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\ **Post date:** [October 15, 2023, 10:57am UTC](https://aiida.discourse.group/t/lose-provenance-inside-workchain/126/5 "2023-10-15T10:57:45Z")
Think it might indeed be useful to have this somewhere in the documentation. Would have been even better as a sort of blog post. We started the idea for this at some point, but it never crystallized. I will have a look if it can be revived. Otherwise I will add it to the docs, but I think it will require some work to add illustrations and code examples to make the concepts clear. ---
**Author:** ![giovannipizzi](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/giovannipizzi/32/17_2.png) [@giovannipizzi](https://aiida.discourse.group/u/giovannipizzi)\ **Post date:** [October 16, 2023, 9:22am UTC](https://aiida.discourse.group/t/lose-provenance-inside-workchain/126/6 "2023-10-16T09:22:05Z")
Indeed, it wouldn’t be bad to revive the blog! We still have the old text (in an obscure branch…) on how to explore MC2D in Quantum Mobile: [https://github.com/aiidateam/aiida-blog/blob/exploredb/source/stories/browse\_discover.rst](https://github.com/aiidateam/aiida-blog/blob/exploredb/source/stories/browse_discover.rst) (and I see there is also a branch by Chris, to check). Sebastiaan, if you want to check where we could host this (I’d try to do the simplest thing possible, even as a new section of the docs, or anything easy to find anyway) we can then discuss how to proceed at the next AiiDA meeting? ---
**Author:** ![Xing](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/xing/32/10_2.png) [@Xing](https://aiida.discourse.group/u/Xing)\ **Post date:** [October 18, 2023, 7:21am UTC](https://aiida.discourse.group/t/lose-provenance-inside-workchain/126/7 "2023-10-18T07:21:19Z")
@sphuber Thanks for the detailed explanation. I agree that we that we should focus on the goal of the calculation, instead of perfect provenance. For the moment, the “level of provenance” is not clear in the doc. So the developer may worry about the unknown risk when they modify the input data. A blog showing the example “level of provenance” with their risk will be very useful.