Architecture

The high level architecture that we need to have has to support many modules/features running in parallel (we fork a single thread per-feature and then a feature forks additional threads). We can imagine we have several modules which are organized by their high-level responsibility and we call those modules, which are effectively code chunks grouped together, features.

Features

Each feature is responsible for a piece of code and has clearly defined responsibilities - we can imagine we have the following features:

  • logging layer

  • monitoring layer

  • network layer

  • blockchain layer

  • ledger layer

  • wallet (backend) layer

cardano-shell-integration

And each of these features have some dependencies, requires some configuration and produces an interface that the user (developer) can use. For example, let’s suppose we want to create a very simple logging/monitoring feature (this example will have one feature which combines both logging and monitoring, but they can be separate).

logging_monitoring_deps

We see that the feature requires a specific configuration and has some global exception handling. The specific configuration can be something really specific to logging, for example. We may need to read the default log level, whether it’s info, debug, warn or something else - we need to define the level of information we want to save in the logs. For that, we require a specific configuration. That configuration is not part of the default configuration we usually parse when we start the cardano-shell - when we parse the initial key configuration, we parse what every feature requires, not something specific for each feature. Then we have the feature ready for usage and we can use the functions it contains. We will restrict this a bit later, but for now, let’s suppose we can use all the functions in the module export. And so, all is well. We are done! But wait! What about the other features? Ok, so I lied, we are not done. We need to able to communicate with the other nodes. What do we need for that? Networking! So we need to construct the networking feature. And what does networking use? It uses loggging! How does that look like?

networking_deps

Again we have a specific configuration for the networking feature (for example the number of surrounding nodes we can communicate with?), we have some global exception handling, and we have a dependency! What is the dependency? It’s the logging/monitoring (sorry for the confusion, I split them up here, since they really seem to be separate) feature. And now we have constructed the networking feature! Are we done? NO! We can communicate with other nodes, but we don’t have any blockchain logic running on the node. So we need to construct the blockchain feature. What are the dependencies for the blockchain feature? Logging, monitoring and networking. And again we have some specific configuration that we may need to read. In this case, it may be the number of blocks per epoch or epoch length or something else.

blockchain_deps

Are we done now? NO! But let’s add just one more example, we can imagine stacking this up for quite some time. We need to construct the ledger feature since we need a way to account for our balances and our transactions and a ton of other stuff. Again we might require logging, monitoring, networking (maybe at this level we might not need to call the networking feature directly, but let’s suppose that we do) and the blockchain. Again we have some specific configuration, for example the cost of the transaction (or something that we use to calculate it), some global exception handling and the dependencies. How does that look?

ledger_dep

Are we done now? NO!

Layers

I lied before. It’s not quite that simple. Let’s go back to the beginning.

We need to be able to test this code easily. And that’s harder than it sounds. When you are not building your code with testing in mind, it becomes very hard to actually test it. So we want to provide some interface towards the features that we can stub out and replace with our test functions. We call those interfaces a layer. We then pass those interfaces around and when we want to replace them, we simply send the replaced interface around. We want to do that in order to clearly separate the features and to enable us to test them in isolation. I hope that makes sense, I don’t want to write the explanation how testing works. A similar approach could be achieved by using a typeclass but it’s not as nice. And this is not good just for testing. It’s good for the project in general - we program towards interfaces and all the client needs to know about are the types and the interface. If we hide the type internals (directly or indirectly) we have a very nice way to build a maintainable project that can change it’s behavior very quickly and the client won’t be affected. So let’s draw some pictures here and make this a bit more clear. Let’s simplify and say we will take a look at only three features - logging, networking and blockchain.

feature_layer_1

When we complete the construction of this logging/monitoring feature, we get the initialized layer, which is the interface toward the feature. It’s essentially a record of functions. Let’s make this a bit more concrete. Let’s say we have an extremely simple interface for logging and all we want our users (developers) to use is this:

logInfo     :: Text -> IO ()
logWarning  :: Text -> IO ()
logDebug    :: Text -> IO ()

feature_layer_2

And we wrap this up in a data structure that is going to represent our interface.

feature_layer_3

So we wrapped our functions in an interface and when we pass it as a dependency to the next feature we can always stub out the functions and see how the feature behaves in isolation. In a way, the produced layer is the result of the feature, and we can pass it down as a dependency to the next feature.

feature_layer_4

We can imagine doing the same for the networking feature - we construct a feature layer that is the interface towards the network feature and we pass it on as a dependency.

feature_layer_5

And you shouldn’t be shocked by the next image. What? They pass it as a dependency? Madness!

feature_layer_6

The only additional thing that you can see in the image is that we pass the logging layer as another dependency as well. So now we can mock out both features and use stubs for both.

Are we done now? NO!

I lied again. I simplified it a bit. If you take a look at the actual layer example that we used, it uses IO. And are we going to use IO on every function? I hope not. What can we do? Well we can keep the type parameter abstract and instantiate it to a proper set of effects later? Sounds like a great idea!

feature_layer_abstract_effect

In the example, we suppose that the interface is abstract (the m type is abstract) and we later on define it to be forall m. (MonadIO m, MonadLog m) => m, so an mtl style effect which has two constraints on it - it can log and do IO, which sounds good. But there is nothing perfect and there are trade-offs everywhere. Even in our nice little example. How? Let me show you. The abstract type is now constrained by two types - MonadIO and MonadLog.

feature_layer_effect_escape_1

So every function in that layer must contain these two constraints. Why? Well because m in our example is the product type of all the effects of all the functions in a layer, right? If we change our logDebug function to contain, say MonadState Text m what can the other functions contain? Yes, they all can contain state. We are allowing functions to use effects that we didn’t intend - isn’t that one of the reasons we use Haskell in the first place? Even worse, seems all our functions are able to do IO, regardless of whether they actually need to use it!

feature_layer_effect_escape_2

But that’s not the worst part! The worst part is yet to come! What happens when we use logging layer in our networking feature function?

feature_layer_effect_escape_3

Yes, now that function must infer the constraints. And if we use logging in one of our functions in the networking layer, what do we get? Yes, IO in all the functions, since we unify all the constrains under one abstract type parameter m. And if the blockchain feature uses the networking layer?

feature_layer_effect_escape_4

It’s spreading! All over our codebase! And if, say, networking layer has more constraints (which probably has), what happens?

feature_layer_effect_escape_5

In the end, the actual working layer on top, say the wallet layer, has no restrictions on the effects. Chaos! We don’t want inheritance, we want composition! That was the promise. Well, we can fix this. The price is hard-to-read type errors if you miss something, but it’s better then the alternative. So we can simply use Rank2Types extension and use a layer that has the constraints defined per function:

data LoggingLayer = LoggingLayer
    { llLogDebug   :: forall m. (MonadIO m) => Text -> m ()
    , llLogInfo    :: forall m. (MonadIO m) => Text -> m ()
    , llLockNonIO  :: forall m. (MonadThrow m) => m ()
    }

And now, we don’t have these issues.

Are we done now? YES!