Reduce the frequency for database updating

[zhuoyuan-liu]

All read requests such as config and query read would update the database about the node info. However, these updates would become a bottleneck for database performance.

I understand that many functions reply on the last seen timestamps, but it would be not scaleable with such heavily written requests to the database. I have two ideas, one is to store these data in Redis/memory and update them periodically. Another is that we process these changes async.

In our test, we only have 3 req/s but the latency between database and osctrl is around 50ms.

Looking forward to hearing from your idea.

[javuto]

I have thought about this several times and I believe combining those two ideas may be the way to go:

• We load nodes in Redis/memory and each field that is now refreshed and updated in the backend, it is up-to-date only in Redis/memory.
• A task is kicked off in the background to keep the backend in sync with what it is in Redis/memory on an interval basis. This can be a goroutine that runs in osctrl-admin or osctrl-tls, like we have others now, and runs every hour or so, with less impact to the backend. Those operations can also be batched.

Thoughts?

[zhuoyuan-liu]

It would definitely increase the scalability, but not sure how far this method would go. I would like to focus on this issue #558 and get some of your feedback. We will also test the osctrl on a larger scale this/next month and provide more feedback/improvement about the performance.