Speed up remove_stale_channels_and_tracking nontrivially

During startup, the lightning protocol forces us to fetch a ton of
gossip for channels where there is a `channel_update` in only one
direction. We then have to wait around a while until we can prune
the crap cause we don't know when the gossip sync has completed.

Sadly, doing a large prune via `remove_stale_channels_and_tracking`
is somewhat slow. Removing a large portion of our graph currently
takes a bit more than 7.5 seconds on an i9-14900K, which can
ultimately ~hang a node with a few less GHz ~forever.

The bulk of this time is in our `IndexedMap` removals, where we
walk the entire `keys` `Vec` to remove the entry, then shift it
down after removing.

Here we shift to a bulk removal model when removing channels, doing
a single `Vec` iterate + shift. This reduces the same test to
around 1.38 seconds on the same hardware.

Author: TheBlueMatt

lightning/src/routing/gossip.rs

@@ -2356,9 +2356,7 @@ where
 			return;
 		}
 		let min_time_unix: u32 = (current_time_unix - STALE_CHANNEL_UPDATE_AGE_LIMIT_SECS) as u32;
-		// Sadly BTreeMap::retain was only stabilized in 1.53 so we can't switch to it for some
-		// time.
-		let mut scids_to_remove = Vec::new();
+		let mut scids_to_remove = new_hash_set();
 		for (scid, info) in channels.unordered_iter_mut() {
 			if info.one_to_two.is_some()
 				&& info.one_to_two.as_ref().unwrap().last_update < min_time_unix
@@ -2382,18 +2380,19 @@ where
 				if announcement_received_timestamp < min_time_unix as u64 {
 					log_gossip!(self.logger, "Removing channel {} because both directional updates are missing and its announcement timestamp {} being below {}",
 						scid, announcement_received_timestamp, min_time_unix);
-					scids_to_remove.push(*scid);
+					scids_to_remove.insert(*scid);
 				}
 			}
 		}
 		if !scids_to_remove.is_empty() {
 			let mut nodes = self.nodes.write().unwrap();
-			for scid in scids_to_remove {
-				let info = channels
-					.remove(&scid)
-					.expect("We just accessed this scid, it should be present");
+			let mut removed_channels_lck = self.removed_channels.lock().unwrap();
+
+			let channels_removed_bulk = channels.remove_fetch_bulk(&scids_to_remove);
+			removed_channels_lck.reserve(channels_removed_bulk.len());
+			for (scid, info) in channels_removed_bulk {
 				self.remove_channel_in_nodes(&mut nodes, &info, scid);
-				self.removed_channels.lock().unwrap().insert(scid, Some(current_time_unix));
+				removed_channels_lck.insert(scid, Some(current_time_unix));
 			}
 		}
 

lightning/src/util/indexed_map.rs

@@ -72,6 +72,18 @@ impl<K: Clone + Hash + Ord, V> IndexedMap<K, V> {
 		ret
 	}
 
+	/// Removes elements with the given `keys` in bulk, returning the set of removed elements.
+	pub fn remove_fetch_bulk(&mut self, keys: &HashSet<K>) -> Vec<(K, V)> {
+		let mut res = Vec::with_capacity(keys.len());
+		for key in keys.iter() {
+			if let Some((k, v)) = self.map.remove_entry(key) {
+				res.push((k, v));
+			}
+		}
+		self.keys.retain(|k| !keys.contains(k));
+		res
+	}
+
 	/// Inserts the given `key`/`value` pair into the map, returning the element that was
 	/// previously stored at the given `key`, if one exists.
 	pub fn insert(&mut self, key: K, value: V) -> Option<V> {