Return transport errors to the caller (#399)

* Make client::InFlightRequests generic over result.

Previously, InFlightRequests required the client response type to be a
server response. However, this prevented injection of non-server
responses: for example, if the client fails to send a request, it should
complete the request with an IO error rather than a server error.

* Gracefully handle client-side send errors.

Previously, a client channel would immediately disconnect when
encountering an error in Transport::try_send. One kind of error that can
occur in try_send is message validation, e.g. validating a message is
not larger than a configured frame size. The problem with shutting down
the client immediately is that debuggability suffers: it can be hard to
understand what caused the client to fail. Also, these errors are not
always fatal, as with frame size limits, so complete shutdown was
extreme.

By bubbling up errors, it's now possible for the caller to
programmatically handle them. For example, the error could be walked
via anyhow::Error:

```
    2023-01-10T02:49:32.528939Z  WARN client: the client failed to send the request

    Caused by:
        0: could not write to the transport
        1: frame size too big
```

* Some follow-up work: right now, read errors will bubble up to all pending RPCs. However, on the write side, only `start_send` bubbles up. `poll_ready`, `poll_flush`, and `poll_close` do not propagate back to pending RPCs. This is probably okay in most circumstances, because fatal write errors likely coincide with fatal read errors, which *do* propagate back to clients. But it might still be worth unifying this logic.

---------

Co-authored-by: Tim Kuehn <tikue@google.com>

example-service/src/client.rs

@@ -26,7 +26,8 @@ async fn main() -> anyhow::Result<()> {
     let flags = Flags::parse();
     init_tracing("Tarpc Example Client")?;
 
-    let transport = tarpc::serde_transport::tcp::connect(flags.server_addr, Json::default);
+    let mut transport = tarpc::serde_transport::tcp::connect(flags.server_addr, Json::default);
+    transport.config_mut().max_frame_length(usize::MAX);
 
     // WorldClient is generated by the service attribute. It has a constructor `new` that takes a
     // config and any Transport as input.
@@ -42,7 +43,10 @@ async fn main() -> anyhow::Result<()> {
     .instrument(tracing::info_span!("Two Hellos"))
     .await;
 
-    tracing::info!("{:?}", hello);
+    match hello {
+        Ok(hello) => tracing::info!("{hello:?}"),
+        Err(e) => tracing::warn!("{:?}", anyhow::Error::from(e)),
+    }
 
     // Let the background span processor finish.
     sleep(Duration::from_micros(1)).await;