Throttle greedy clients in orderer Broadcast API

[yacovm]

This commit implements a shared rate limiter for connections authenticated with mutual TLS. The throttling and rate limiting is applied at both the Authority Key Identifier and Subject Key Identifier level, which guarantees no single organization or client can bombard the orderer with endless transactions on the expense of others.

The effective rate is divided across all clients and orgs for a time period specified by InactivityTimeout.

By default this feature is turned off.

[pfi79]

@yacovm
Why didn't you use an off-the-shelf library like golang.org/x/time/rate?

[yacovm]

@yacovm Why didn't you use an off-the-shelf library like golang.org/x/time/rate?

I am not sure it fits our need. I need a dynamic rate limiter that limits on a per client basis and the budget is adjusted dynamically according to how many clients are connected.

[pfi79]

@yacovm Why didn't you use an off-the-shelf library like golang.org/x/time/rate?

I am not sure it fits our need. I need a dynamic rate limiter that limits on a per client basis and the budget is adjusted dynamically according to how many clients are connected.

I used it for dynamic loading. Conditionally: the TPS of the system had to be X and had to be divided by all nodes. Nodes could crash and restart. And the throughput of the system should not change.

[C0rWin]

There are already rate limiters implemented for Endorsering, Deliver, and Gateway services; wouldn't it be better to extend its usage for ordering service as well? See here: #647 . I understand the existing approach probably doesn't provide the desired granularity, though still, even putting rate limits with the grpc interceptor is a little bit cleaner approach rather than extending the broadcast handler.

[C0rWin]

@yacovm, btw, do you also intend to add metrics to expose limiter budget utilization so operators can trace down setup and alerts to notify whenever the limit is exhausted per client or organization? Now, where the model is to use gateway service, meaning peers are collecting endorsements and submitting transactions, this might be crucial.

[yacovm]

@yacovm Why didn't you use an off-the-shelf library like golang.org/x/time/rate?

I am not sure it fits our need. I need a dynamic rate limiter that limits on a per client basis and the budget is adjusted dynamically according to how many clients are connected.

I used it for dynamic loading. Conditionally: the TPS of the system had to be X and had to be divided by all nodes. Nodes could crash and restart. And the throughput of the system should not change.

I think you're talking about sending transactions and not receiving them, am I mistaken?

[yacovm]

There are already rate limiters implemented for Endorsering, Deliver, and Gateway services; wouldn't it be better to extend its usage for ordering service as well? See here: #647 . I understand the existing approach probably doesn't provide the desired granularity, though still, even putting rate limits with the grpc interceptor is a little bit cleaner approach rather than extending the broadcast handler.

They are not rate limiters, they are simply locks that limit too many gRPC streams being opened. This does not address the problem solved here.

I think what I could have done alternatively was to make a proxy Broadcast(srv ab.AtomicBroadcast_BroadcastServer) wrapper which throttles right after calling Recv() and then forwards and returns the backend Recv() of the gRPC stream, and then the broadcast.go would have stayed intact.

@C0rWin I made the above modification, now the broadcast service implementation remains as-is.

[yacovm]

@yacovm, btw, do you also intend to add metrics to expose limiter budget utilization so operators can trace down setup and alerts to notify whenever the limit is exhausted per client or organization? Now, where the model is to use gateway service, meaning peers are collecting endorsements and submitting transactions, this might be crucial.

The idea is cool but I have no intention of doing this, as this piece of work is not meant to be used in this manner.
This is only to prevent a client from spamming an orderer with too many transactions.
As long as you're an "average" client and you don't over saturate your budget on a timely manner, you should be good.