Web3Signer support for VC (#2522)

[EIP-3030]: https://eips.ethereum.org/EIPS/eip-3030
[Web3Signer]: https://consensys.github.io/web3signer/web3signer-eth2.html

## Issue Addressed

Resolves #2498

## Proposed Changes

Allows the VC to call out to a [Web3Signer] remote signer to obtain signatures.


## Additional Info

### Making Signing Functions `async`

To allow remote signing, I needed to make all the signing functions `async`. This caused a bit of noise where I had to convert iterators into `for` loops.

In `duties_service.rs` there was a particularly tricky case where we couldn't hold a write-lock across an `await`, so I had to first take a read-lock, then grab a write-lock.

### Move Signing from Core Executor

Whilst implementing this feature, I noticed that we signing was happening on the core tokio executor. I suspect this was causing the executor to temporarily lock and occasionally trigger some HTTP timeouts (and potentially SQL pool timeouts, but I can't verify this). Since moving all signing into blocking tokio tasks, I noticed a distinct drop in the "atttestations_http_get" metric on a Prater node:

![http_get_times](https://user-images.githubusercontent.com/6660660/132143737-82fd3836-2e7e-445b-a143-cb347783baad.png)

I think this graph indicates that freeing the core executor allows the VC to operate more smoothly.

### Refactor TaskExecutor

I noticed that the `TaskExecutor::spawn_blocking_handle` function would fail to spawn tasks if it were unable to obtain handles to some metrics (this can happen if the same metric is defined twice). It seemed that a more sensible approach would be to keep spawning tasks, but without metrics. To that end, I refactored the function so that it would still function without metrics. There are no other changes made.

## TODO

- [x] Restructure to support multiple signing methods.
- [x] Add calls to remote signer from VC.
- [x] Documentation
- [x] Test all endpoints
- [x] Test HTTPS certificate
- [x] Allow adding remote signer validators via the API
- [x] Add Altair support via [21.8.1-rc1](https://github.com/ConsenSys/web3signer/releases/tag/21.8.1-rc1)
- [x] Create issue to start using latest version of web3signer. (See #2570)

## Notes

- ~~Web3Signer doesn't yet support the Altair fork for Prater. See Consensys/web3signer#423
- ~~There is not yet a release of Web3Signer which supports Altair blocks. See Consensys/web3signer#391

Author: paulhauner

Cargo.lock

@@ -76,6 +76,7 @@ members = [
     "testing/node_test_rig",
     "testing/simulator",
     "testing/state_transition_vectors",
+    "testing/web3signer_tests",
 
     "validator_client",
     "validator_client/slashing_protection",

Cargo.toml

@@ -32,6 +32,7 @@
 * [Advanced Usage](./advanced.md)
     * [Custom Data Directories](./advanced-datadir.md)
     * [Validator Graffiti](./graffiti.md)
+    * [Remote Signing with Web3Signer](./validator-web3signer.md)
     * [Database Configuration](./advanced_database.md)
     * [Advanced Networking](./advanced_networking.md)
     * [Running a Slasher](./slasher.md)

book/src/SUMMARY.md

@@ -434,3 +434,43 @@ Typical Responses | 200
     ]
 }
 ```
+
+## `POST /lighthouse/validators/web3signer`
+
+Create any number of new validators, all of which will refer to a
+[Web3Signer](https://docs.web3signer.consensys.net/en/latest/) server for signing.
+
+### HTTP Specification
+
+| Property | Specification |
+| --- |--- |
+Path | `/lighthouse/validators/web3signer`
+Method | POST
+Required Headers | [`Authorization`](./api-vc-auth-header.md)
+Typical Responses | 200, 400
+
+### Example Request Body
+
+```json
+[
+    {
+        "enable": true,
+        "description": "validator_one",
+        "graffiti": "Mr F was here",
+        "voting_public_key": "0xa062f95fee747144d5e511940624bc6546509eeaeae9383257a9c43e7ddc58c17c2bab4ae62053122184c381b90db380",
+        "url": "http://path-to-web3signer.com",
+        "root_certificate_path": "/path/on/vc/filesystem/to/certificate.pem",
+        "request_timeout_ms": 12000
+    }
+]
+```
+
+The following fields may be omitted or nullified to obtain default values:
+
+- `graffiti`
+- `root_certificate_path`
+- `request_timeout_ms`
+
+### Example Response Body
+
+*No data is included in the response body.*

book/src/api-vc-endpoints.md

@@ -0,0 +1,56 @@
+# Remote Signing with Web3Signer
+
+[Web3Signer]: https://docs.web3signer.consensys.net/en/latest/
+[Consensys]: https://github.com/ConsenSys/
+[Teku]: https://github.com/consensys/teku
+
+[Web3Signer] is a tool by Consensys which allows *remote signing*. Remote signing is when a
+Validator Client (VC) out-sources the signing of messages to remote server (e.g., via HTTPS). This
+means that the VC does not hold the validator private keys.
+
+## Warnings
+
+Using a remote signer comes with risks, please read the following two warnings before proceeding:
+
+### Remote signing is complex and risky
+
+Remote signing is generally only desirable for enterprise users or users with unique security
+requirements. Most users will find the separation between the Beacon Node (BN) and VC to be
+sufficient *without* introducing a remote signer.
+
+**Using a remote signer introduces a new set of security and slashing risks and should only be
+undertaken by advanced users who fully understand the risks.**
+
+### Web3Signer is not maintained by Lighthouse
+
+The [Web3Signer] tool is maintained by [Consensys], the same team that maintains [Teku]. The
+Lighthouse team (Sigma Prime) does not maintain Web3Signer or make any guarantees about its safety
+or effectiveness.
+
+## Usage
+
+A remote signing validator is added to Lighthouse in much the same way as one that uses a local
+keystore, via the [`validator_definitions.yml`](./validator-management.md) file or via the `POST
+/lighthouse/validators/web3signer` API endpoint.
+
+Here is an example of a `validator_definitions.yml` file containing one validator which uses a
+remote signer:
+
+```yaml
+---
+- enabled: true
+  voting_public_key: "0xa5566f9ec3c6e1fdf362634ebec9ef7aceb0e460e5079714808388e5d48f4ae1e12897fed1bea951c17fa389d511e477"
+  type: web3signer
+  url: "https://my-remote-signer.com:1234"
+  root_certificate_path: /home/paul/my-certificates/my-remote-signer.pem
+```
+
+When using this file, the Lighthouse VC will perform duties for the `0xa5566..` validator and defer
+to the `https://my-remote-signer.com:1234` server to obtain any signatures. It will load a
+"self-signed" SSL certificate from `/home/paul/my-certificates/my-remote-signer.pem` (on the
+filesystem of the VC) to encrypt the communications between the VC and Web3Signer.
+
+> The `request_timeout_ms` key can also be specified. Use this key to override the default timeout
+> with a new timeout in milliseconds. This is the timeout before requests to Web3Signer are
+> considered to be failures. Setting a value that it too-long may create contention and late duties
+> in the VC.  Setting it too short will result in failed signatures and therefore missed duties.

book/src/validator-web3signer.md

@@ -61,6 +61,21 @@ pub enum SigningDefinition {
         #[serde(skip_serializing_if = "Option::is_none")]
         voting_keystore_password: Option<ZeroizeString>,
     },
+    /// A validator that defers to a Web3Signer HTTP server for signing.
+    ///
+    /// https://github.com/ConsenSys/web3signer
+    #[serde(rename = "web3signer")]
+    Web3Signer {
+        url: String,
+        /// Path to a .pem file.
+        #[serde(skip_serializing_if = "Option::is_none")]
+        root_certificate_path: Option<PathBuf>,
+        /// Specifies a request timeout.
+        ///
+        /// The timeout is applied from when the request starts connecting until the response body has finished.
+        #[serde(skip_serializing_if = "Option::is_none")]
+        request_timeout_ms: Option<u64>,
+    },
 }
 
 /// A validator that may be initialized by this validator client.
@@ -116,6 +131,12 @@ impl ValidatorDefinition {
 #[derive(Default, Serialize, Deserialize)]
 pub struct ValidatorDefinitions(Vec<ValidatorDefinition>);
 
+impl From<Vec<ValidatorDefinition>> for ValidatorDefinitions {
+    fn from(vec: Vec<ValidatorDefinition>) -> Self {
+        Self(vec)
+    }
+}
+
 impl ValidatorDefinitions {
     /// Open an existing file or create a new, empty one if it does not exist.
     pub fn open_or_create<P: AsRef<Path>>(validators_dir: P) -> Result<Self, Error> {
@@ -167,11 +188,13 @@ impl ValidatorDefinitions {
         let known_paths: HashSet<&PathBuf> = self
             .0
             .iter()
-            .map(|def| match &def.signing_definition {
+            .filter_map(|def| match &def.signing_definition {
                 SigningDefinition::LocalKeystore {
                     voting_keystore_path,
                     ..
-                } => voting_keystore_path,
+                } => Some(voting_keystore_path),
+                // A Web3Signer validator does not use a local keystore file.
+                SigningDefinition::Web3Signer { .. } => None,
             })
             .collect();
 

common/account_utils/src/validator_definitions.rs

@@ -313,6 +313,22 @@ impl ValidatorClientHttpClient {
         self.post(path, &request).await
     }
 
+    /// `POST lighthouse/validators/web3signer`
+    pub async fn post_lighthouse_validators_web3signer(
+        &self,
+        request: &[Web3SignerValidatorRequest],
+    ) -> Result<GenericResponse<ValidatorData>, Error> {
+        let mut path = self.server.full.clone();
+
+        path.path_segments_mut()
+            .map_err(|()| Error::InvalidUrl(self.server.clone()))?
+            .push("lighthouse")
+            .push("validators")
+            .push("web3signer");
+
+        self.post(path, &request).await
+    }
+
     /// `PATCH lighthouse/validators/{validator_pubkey}`
     pub async fn patch_lighthouse_validators(
         &self,

common/eth2/src/lighthouse_vc/http_client.rs

@@ -2,6 +2,7 @@ use account_utils::ZeroizeString;
 use eth2_keystore::Keystore;
 use graffiti::GraffitiString;
 use serde::{Deserialize, Serialize};
+use std::path::PathBuf;
 
 pub use crate::lighthouse::Health;
 pub use crate::types::{GenericResponse, VersionData};
@@ -64,3 +65,20 @@ pub struct KeystoreValidatorsPostRequest {
     pub keystore: Keystore,
     pub graffiti: Option<GraffitiString>,
 }
+
+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
+pub struct Web3SignerValidatorRequest {
+    pub enable: bool,
+    pub description: String,
+    #[serde(default)]
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub graffiti: Option<GraffitiString>,
+    pub voting_public_key: PublicKey,
+    pub url: String,
+    #[serde(default)]
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub root_certificate_path: Option<PathBuf>,
+    #[serde(default)]
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub request_timeout_ms: Option<u64>,
+}

common/eth2/src/lighthouse_vc/types.rs

@@ -283,6 +283,18 @@ pub fn set_gauge_vec(int_gauge_vec: &Result<IntGaugeVec>, name: &[&str], value:
     }
 }
 
+pub fn inc_gauge_vec(int_gauge_vec: &Result<IntGaugeVec>, name: &[&str]) {
+    if let Some(gauge) = get_int_gauge(int_gauge_vec, name) {
+        gauge.inc();
+    }
+}
+
+pub fn dec_gauge_vec(int_gauge_vec: &Result<IntGaugeVec>, name: &[&str]) {
+    if let Some(gauge) = get_int_gauge(int_gauge_vec, name) {
+        gauge.dec();
+    }
+}
+
 pub fn set_gauge(gauge: &Result<IntGauge>, value: i64) {
     if let Ok(gauge) = gauge {
         gauge.set(value);

common/lighthouse_metrics/src/lib.rs

@@ -237,39 +237,33 @@ impl TaskExecutor {
     {
         let log = self.log.clone();
 
-        if let Some(metric) = metrics::get_histogram(&metrics::BLOCKING_TASKS_HISTOGRAM, &[name]) {
-            if let Some(int_gauge) = metrics::get_int_gauge(&metrics::BLOCKING_TASKS_COUNT, &[name])
-            {
-                let int_gauge_1 = int_gauge;
-                let timer = metric.start_timer();
-                let join_handle = if let Some(runtime) = self.runtime.upgrade() {
-                    runtime.spawn_blocking(task)
-                } else {
-                    debug!(self.log, "Couldn't spawn task. Runtime shutting down");
-                    return None;
-                };
+        let timer = metrics::start_timer_vec(&metrics::BLOCKING_TASKS_HISTOGRAM, &[name]);
+        metrics::inc_gauge_vec(&metrics::BLOCKING_TASKS_COUNT, &[name]);
 
-                Some(async move {
-                    let result = match join_handle.await {
-                        Ok(result) => {
-                            trace!(log, "Blocking task completed"; "task" => name);
-                            Ok(result)
-                        }
-                        Err(e) => {
-                            debug!(log, "Blocking task ended unexpectedly"; "error" => %e);
-                            Err(e)
-                        }
-                    };
-                    timer.observe_duration();
-                    int_gauge_1.dec();
-                    result
-                })
-            } else {
-                None
-            }
+        let join_handle = if let Some(runtime) = self.runtime.upgrade() {
+            runtime.spawn_blocking(task)
         } else {
-            None
-        }
+            debug!(self.log, "Couldn't spawn task. Runtime shutting down");
+            return None;
+        };
+
+        let future = async move {
+            let result = match join_handle.await {
+                Ok(result) => {
+                    trace!(log, "Blocking task completed"; "task" => name);
+                    Ok(result)
+                }
+                Err(e) => {
+                    debug!(log, "Blocking task ended unexpectedly"; "error" => %e);
+                    Err(e)
+                }
+            };
+            drop(timer);
+            metrics::dec_gauge_vec(&metrics::BLOCKING_TASKS_COUNT, &[name]);
+            result
+        };
+
+        Some(future)
     }
 
     pub fn runtime(&self) -> Weak<Runtime> {