State of The Network | Jan 2nd 2022

Latest update below marked with “**Update ])

​

https://preview.redd.it/j0ijovimfe981.png?width=1030&format=png&auto=webp&s=4eb4f31c2b033e29a1c138e843a85b4737a7ff53

We feel it's important to keep the community informed with communication and transparency, while accepting constructive feedback and listening to your concerns. The on-going RPC issues have our attention, and we will not be treating this lightly. Improvements to the infrastructure are underway and with more to come. Some of the upcoming work includes:

a. RPC Change #1 (completed)

Problematic RPC nodes were redirected to functional regions, restarted, confirmed as caught up, then placed back into service This is a workaround. Items below are long-term solutions.

b. Arbitrage Bot Transactions & Gas Price (upcoming)

Nodes are receiving an urgent update to increase recommended gas price from 10 gwei to 30 gwei and minimum gas price from 1 gwei to 30 gwei. The fee increase will combat the arbitrage bots spamming the network with transactions. No changes should be required from anyone in the ecosystem using our recommended minimums. Any one-off dapps using their own gas fees will need users to manually adjust.

*[Update Jan 3]) When: We want this to be as smooth of a transition as possible for everyone in our ecosystem. Dates will be published soon, giving developers an opportunity to update their apps if necessary before anything is modified.

c. RPC Change #2 | Rate Limiting (partially completed)

RPCs used specifically for staking data are now rate limited. These requests are very taxing, and could drag down all RPC nodes on the network, causing them to fall out-of-sync. We noticed a correlation between the two some hours ago.

*[Update Jan 3]) When: Rate limits are being tested on a few RPC nodes with no issues. Roll out is is postponed as our focus is on (d) below.

**Update Jan 5]) We have two RPC methods bogging down the nodes with one method processing 5,000 queries in 5 minutes and another processing 2.5k queries in 5 minutes. We're adding rate limiters to those RPCs and will be pushing it tonight.

Upcoming fix: https://github.com/harmony-one/harmony/pull/3971

**Update Jan 6]) The rate limiting above has been applied to most RPC regions and we're seeing a lower number of requests processed. Next, we will be rate limiting another RPC which appears to be excessively used and resulting in poor node performance, doEVMcall. It's being merged into PR #3971 along with above limiters.

d. RPC Change #3 | Sync Status Checks (partially completed)

Our load balancers query RPC nodes to determine their health. When a node is out-of-sync, and to avoid impact, the load balancers are expected to stop sending queries in their direction. However, we're finding queries are still sent to out-of-sync nodes despite their failed health check. We developed a possible solution to this problem and are in the process of testing the code.

**Update Jan 5]) This change has been applied to all our nodes and we're seeing a significant improvement in their status report back to the Load Balancers. This doesn't mean all issues are resolved (see (c) above) but simply that we've successfully resolved this specific issue which created the universal “RPC is down” symptom.

**Update Jan 6]) The sync checks against RPC nodes appears to be failing sporadically, leading to queries still reaching out-of-sync endpoints. We are working diligently on determining root cause and implement a solution. In the meantime, we are creating a script which will detect when a node is reporting falsely and take action to resolve it. This will help maintain a much more stable experience for the community.

We all thank you for your patience and support as we continue to resolve these issues. Your experience and support means a great deal, and our team and community will work diligently to improve the network for everyone.

Harmony Foundation πŸ’™

+++++++++++++++++++++++++++++++++++++++++++

Workaround

If you're impacted, please follow the steps documented below and configure your Metamask to use the Pocket network. The team at Pocket has made great advancements over the past few weeks with Harmony RPC nodes and appears to be handling traffic very well.

Click on the Networks drop-down menu, then press Custom RPC

Under the Network Name field, type Harmony Mainnet 0 POKT Portal

Within the New RPC URL field, copy and paste this endpoint URL https://harmony-0-rpc.gateway.pokt.network

Put the decimal 1666600000 in the ChainID field

Write ONE as the Symbol

Add https://explorer.harmony.one as the Block Explorer URL

Don’t forget to save

+++++++++++++++++++++++++++++++++++++++++++

State of The Network | Jan 2nd 2022

17 thoughts on “State of The Network | Jan 2nd 2022”

  1. We encourage quality content intended to help and educate the community. If you have questions or concerns about the subreddit, send us a message and say hello! Cheers and enjoy. Note: Beware of scammers attempting to assist you via direct message. Be wary of any links sent to you via direct message asking to connect your wallet and inputting your seed phrase.

    I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

  2. Note that our explorer is behind by approximately 5-hours until it’s fully synchronized. The full sync task should complete by 2AM PST. We are also in talks with Etherscan and evaluating their options for outsourcing the explorer, but no guarantees can be made at this time.

    Also a quick mention about the bump in gas fee β€” this too is a temporary solution with the more permanent solution being cross shard contract support, allowing dapps to use other the other shards. This is under development.

    Edit – Jan 5th – See updates above marked [Update Jan 5]. We have identified an issue which is returning similar symptoms to those caused by item (d). Fix linked above as well and will be going in tonight.

    Edit – Jan 6th – Additional updates made above marked [Update Jan 6]. We are working diligently and will continue keeping everyone posted as updates are available.

  3. I’m frustrated by the Harmony one website. I recall that it was much easier to navigate months ago. A search box is needed, imo. I’m a retail investor who bought in last spring. Today, I was looking for some wallet and staking information, and I would of liked updates on some earlier hackathon winners. But your present site makes it very difficult and frustrating to do.

  4. Brilliant, thanks for the detailed update. One concern I’ve got seems to be the dependency on Amazon AWS. Are there redundant nodes on other hosts (e.g. Azure cloud)?

  5. What qualifies as a an RPC call for staking data?

    Does a balanceOf check on a HRC20 contract count?

  6. Thank you for the update.

    I think it’s really important to focus on fixing all the network issues asap, even if it means delaying some upcoming features.

    A lot of people are jumping on the Harmony ONE ship due to DFK and all the other cool projects, and it’s a pity to have a bad first experience with the network.

  7. The communication and response to these issues is top notch guys, thanks for the transparency and the hard work!

  8. Thank you for giving us an update and a look at potential upcoming changes!

    Is there any current or in the works plan to figure out a way to disincentivize the arb bot behavior that is adding to this problem like /u/anykeyh suggested? It seems like even with cross shard support a reduction in gas fees would just make bots fill up shards with transactions as well

  9. I’m curious, how do you know that raising the gas fee will help to stop the bots?

  10. This gas fee increase is severely underrated. It’s probably the best thing that has happened to the network in a year.

Leave a Comment