State of The Network | Jan 2nd 2022

Latest update below marked with “**Update ])

https://preview.redd.it/j0ijovimfe981.png?width=1030&format=png&auto=webp&s=4eb4f31c2b033e29a1c138e843a85b4737a7ff53

We feel it's important to keep the community informed with communication and transparency, while accepting constructive feedback and listening to your concerns. The on-going RPC issues have our attention, and we will not be treating this lightly. Improvements to the infrastructure are underway and with more to come. Some of the upcoming work includes:

a. RPC Change #1 (completed)

Problematic RPC nodes were redirected to functional regions, restarted, confirmed as caught up, then placed back into service This is a workaround. Items below are long-term solutions.

b. Arbitrage Bot Transactions & Gas Price (upcoming)

Nodes are receiving an urgent update to increase recommended gas price from 10 gwei to 30 gwei and minimum gas price from 1 gwei to 30 gwei. The fee increase will combat the arbitrage bots spamming the network with transactions. No changes should be required from anyone in the ecosystem using our recommended minimums. Any one-off dapps using their own gas fees will need users to manually adjust.

*[Update Jan 3]) When: We want this to be as smooth of a transition as possible for everyone in our ecosystem. Dates will be published soon, giving developers an opportunity to update their apps if necessary before anything is modified.

c. RPC Change #2 | Rate Limiting (partially completed)

RPCs used specifically for staking data are now rate limited. These requests are very taxing, and could drag down all RPC nodes on the network, causing them to fall out-of-sync. We noticed a correlation between the two some hours ago.

*[Update Jan 3]) When: Rate limits are being tested on a few RPC nodes with no issues. Roll out is is postponed as our focus is on (d) below.

**Update Jan 5]) We have two RPC methods bogging down the nodes with one method processing 5,000 queries in 5 minutes and another processing 2.5k queries in 5 minutes. We're adding rate limiters to those RPCs and will be pushing it tonight.

Upcoming fix: https://github.com/harmony-one/harmony/pull/3971

**Update Jan 6]) The rate limiting above has been applied to most RPC regions and we're seeing a lower number of requests processed. Next, we will be rate limiting another RPC which appears to be excessively used and resulting in poor node performance, doEVMcall. It's being merged into PR #3971 along with above limiters.

d. RPC Change #3 | Sync Status Checks (partially completed)

Our load balancers query RPC nodes to determine their health. When a node is out-of-sync, and to avoid impact, the load balancers are expected to stop sending queries in their direction. However, we're finding queries are still sent to out-of-sync nodes despite their failed health check. We developed a possible solution to this problem and are in the process of testing the code.

**Update Jan 5]) This change has been applied to all our nodes and we're seeing a significant improvement in their status report back to the Load Balancers. This doesn't mean all issues are resolved (see (c) above) but simply that we've successfully resolved this specific issue which created the universal “RPC is down” symptom.

**Update Jan 6]) The sync checks against RPC nodes appears to be failing sporadically, leading to queries still reaching out-of-sync endpoints. We are working diligently on determining root cause and implement a solution. In the meantime, we are creating a script which will detect when a node is reporting falsely and take action to resolve it. This will help maintain a much more stable experience for the community.

We all thank you for your patience and support as we continue to resolve these issues. Your experience and support means a great deal, and our team and community will work diligently to improve the network for everyone.

Harmony Foundation 💙

+++++++++++++++++++++++++++++++++++++++++++

Workaround

If you're impacted, please follow the steps documented below and configure your Metamask to use the Pocket network. The team at Pocket has made great advancements over the past few weeks with Harmony RPC nodes and appears to be handling traffic very well.

Click on the Networks drop-down menu, then press Custom RPC

Under the Network Name field, type Harmony Mainnet 0 POKT Portal

Within the New RPC URL field, copy and paste this endpoint URL https://harmony-0-rpc.gateway.pokt.network

Put the decimal 1666600000 in the ChainID field

Write ONE as the Symbol

Add https://explorer.harmony.one as the Block Explorer URL

Don’t forget to save

+++++++++++++++++++++++++++++++++++++++++++

State of The Network | Jan 2nd 2022