what's the meaning of IO error when running cachebench

[davyjang1]

I temporarily can only one host with 16GB memory, so use the following the test config
my goal is to only measure if one FDP enabled disk can keep WAF =~ 1, and I don't know everything of Cachelib
add I met the below error and there is no IO on disk for a long time

{
  "cache_config": {
    "cacheSizeMB": 12000,
    "cacheDir": "/root/cachelib_metadata-1",
    "allocFactor": 1.08,
    "maxAllocSize": 524288,
    "minAllocSize": 64,
    "navyReaderThreads": 72,
    "navyWriterThreads": 36,
    "nvmCachePaths": ["/dev/nvme0n1"
                ],
    "nvmCacheSizeMB" : 878700,
    "writeAmpDeviceList": ["nvme0n1"
                ],
    "navyBigHashBucketSize": 4096,
    "navyBigHashSizePct": 4,
    "navySmallItemMaxSize": 640,
    "navySegmentedFifoSegmentRatio": [
                        1.0
                ],
    "navyHitsReinsertionThreshold": 1,
    "navyBlockSize": 4096,
    "deviceMaxWriteSize": 262144,
    "nvmAdmissionRetentionTimeThreshold": 7200,
    "navyParcelMemoryMB": 256,
    "enableChainedItem": true,
    "htBucketPower": 29,
    "navyQDepth": 1,
    "navyEnableIoUring": true,
    "deviceEnableFDP": false,
    "moveOnSlabRelease": false,
    "poolRebalanceIntervalSec": 2,
    "poolResizeIntervalSec": 2,
    "rebalanceStrategy": "hits"
        },
  "test_config": {
    "opRatePerSec": 1000000,
    "opRateBurstSize": 200,
    "enableLookaside": false,
    "generator": "replay",
    "replayGeneratorConfig": {
      "ampFactor": 200
                },
    "repeatTraceReplay": true,
    "repeatOpCount" : true,
    "onlySetIfMiss" : false,
    "numOps": 100000000000,
    "numThreads": 10,
    "prepopulateCache": true,
    "traceFileNames": [
      "/home/dahui/Documents/cachelib_test/kvcache_traces_1.csv",
      "/home/dahui/Documents/cachelib_test/kvcache_traces_2.csv",
      "/home/dahui/Documents/cachelib_test/kvcache_traces_3.csv",
      "/home/dahui/Documents/cachelib_test/kvcache_traces_4.csv",
      "/home/dahui/Documents/cachelib_test/kvcache_traces_5.csv"
                ]
        }
}
 

#0  facebook::cachelib::navy::(anonymous namespace)::IOOp::done (this=0x7ff898011ee0, len=-22)                                                  [0/1701]
    at /home/dahui/Documents/CacheLib/cachelib/navy/common/Device.cpp:578
#1  0x0000555555d81cf5 in facebook::cachelib::navy::(anonymous namespace)::AsyncIoContext::handleCompletion (this=0x7ff8a0009d10, completed=...)
    at /home/dahui/Documents/CacheLib/cachelib/navy/common/Device.cpp:849
#2  0x0000555555d8275e in facebook::cachelib::navy::(anonymous namespace)::AsyncIoContext::submitIo (this=0x7ff8a0009d10, op=...)
    at /home/dahui/Documents/CacheLib/cachelib/navy/common/Device.cpp:886
#3  0x0000555555d7fa8c in facebook::cachelib::navy::(anonymous namespace)::IoContext::submitReq (this=0x7ff8a0009d10,
    req=std::shared_ptr<facebook::cachelib::navy::(anonymous namespace)::IOReq> (use count 2, weak count 0) = {...})
    at /home/dahui/Documents/CacheLib/cachelib/navy/common/Device.cpp:732
#4  0x0000555555d7f726 in facebook::cachelib::navy::(anonymous namespace)::IoContext::submitRead(const std::vector<folly::File, std::allocator<folly::File> > &, uint32_t, uint64_t, uint32_t, void *, std::function<void(double)>) (this=0x7ff8a0009d10, fvec=std::vector of length 1, capacity 1 = {...},
    stripeSize=0, offset=888384618496, size=4096, data=0x7ff758002000, trackIOOpDeviceLatency=...)
    at /home/dahui/Documents/CacheLib/cachelib/navy/common/Device.cpp:709
#5  0x0000555555d84e13 in facebook::cachelib::navy::(anonymous namespace)::FileDevice::readImpl (this=0x5555563ff9f0, offset=888384618496, size=4096,
    value=0x7ff758002000) at /home/dahui/Documents/CacheLib/cachelib/navy/common/Device.cpp:1006
#6  0x0000555555d7c9c1 in facebook::cachelib::navy::Device::readInternal (this=0x5555563ff9f0, offset=888384618496, size=4096, value=0x7ff758002000)
    at /home/dahui/Documents/CacheLib/cachelib/navy/common/Device.cpp:480
#7  0x0000555555d7d438 in facebook::cachelib::navy::Device::read (this=0x5555563ff9f0, offset=888384618496, size=4096, value=0x7ff758002000)
    at /home/dahui/Documents/CacheLib/cachelib/navy/common/Device.cpp:534
#8  0x0000555555ea228c in facebook::cachelib::navy::BigHash::readBucket (this=0x555556d90c60, bid=...)
    at /home/dahui/Documents/CacheLib/cachelib/navy/bighash/BigHash.cpp:556
#9  0x0000555555ea042a in facebook::cachelib::navy::BigHash::insert (this=0x555556d90c60, hk=..., value=...)
    at /home/dahui/Documents/CacheLib/cachelib/navy/bighash/BigHash.cpp:318
#10 0x0000555555f1b48e in facebook::cachelib::navy::EnginePair::insertInternal (this=0x5555579e9a50, hk=..., value=...,
    skipInsertion=@0x7fffee0635a8: false) at /home/dahui/Documents/CacheLib/cachelib/navy/engine/EnginePair.cpp:82
#11 0x0000555555f1b7a7 in operator() (__closure=0x7fffee063530) at /home/dahui/Documents/CacheLib/cachelib/navy/engine/EnginePair.cpp:119
#12 0x0000555555f1d5d0 in folly::detail::function::call_<facebook::cachelib::navy::EnginePair::scheduleInsert(facebook::cachelib::HashedKey, facebook::cachelib::navy::BufferView, facebook::cachelib::navy::InsertCallback)::<lambda()>, false, false, facebook::cachelib::navy::JobExitCode>(folly::detail::fu
nction::Data &) (p=...) at /home/dahui/Documents/CacheLib/opt/cachelib/include/folly/Function.h:343
#13 0x0000555555e09667 in folly::detail::function::FunctionTraits<facebook::cachelib::navy::JobExitCode ()>::operator()() (this=0x7ff89003ca70)
    at /home/dahui/Documents/CacheLib/opt/cachelib/include/folly/Function.h:370
#14 0x0000555555e06a95 in operator() (__closure=0x7ff89003ca60)
#15 0x0000555555e080b0 in folly::detail::function::call_<facebook::cachelib::navy::OrderedThreadPoolJobScheduler::scheduleJobLocked(JobParams, uint64_t)::<lambda()>, false, false, facebook::cachelib::navy::JobExitCode>(folly::detail::function::Data &) (p=...)
    at /home/dahui/Documents/CacheLib/opt/cachelib/include/folly/Function.h:343
#16 0x0000555555e09667 in folly::detail::function::FunctionTraits<facebook::cachelib::navy::JobExitCode ()>::operator()() (this=0x7ff8d9798fc0)
    at /home/dahui/Documents/CacheLib/opt/cachelib/include/folly/Function.h:370
#17 0x0000555555e0d4d2 in operator() (__closure=0x7ff8d9798d38) at /home/dahui/Documents/CacheLib/cachelib/navy/scheduler/ThreadPoolJobQueue.cpp:115
#18 0x0000555555e0d9d8 in facebook::cachelib::navy::JobQueue::runJob (this=0x555556d897d0, entry=...)
    at /home/dahui/Documents/CacheLib/cachelib/navy/scheduler/ThreadPoolJobQueue.cpp:127
#19 0x0000555555e0d0b1 in facebook::cachelib::navy::JobQueue::process (this=0x555556d897d0)
    at /home/dahui/Documents/CacheLib/cachelib/navy/scheduler/ThreadPoolJobQueue.cpp:84
#20 0x0000555555e046b8 in operator() (__closure=0x555556d89af8) at /home/dahui/Documents/CacheLib/cachelib/navy/scheduler/ThreadPoolJobScheduler.cpp:48
#21 0x0000555555e085e4 in std::__invoke_impl<void, facebook::cachelib::navy::ThreadPoolExecutor::ThreadPoolExecutor(uint32_t, folly::StringPiece)::<lambda()> >(std::__invoke_other, struct {...} &&) (__f=...) at /usr/include/c++/13/bits/invoke.h:61
#22 0x0000555555e085a7 in std::__invoke<facebook::cachelib::navy::ThreadPoolExecutor::ThreadPoolExecutor(uint32_t, folly::StringPiece)::<lambda()> >(struct {...} &&) (__fn=...) at /usr/include/c++/13/bits/invoke.h:96
#23 0x0000555555e08554 in std::thread::_Invoker<std::tuple<facebook::cachelib::navy::ThreadPoolExecutor::ThreadPoolExecutor(uint32_t, folly::StringPiece)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x555556d89af8) at /usr/include/c++/13/bits/std_thread.h:292
#24 0x0000555555e08528 in std::thread::_Invoker<std::tuple<facebook::cachelib::navy::ThreadPoolExecutor::ThreadPoolExecutor(uint32_t, folly::StringPiece)::<lambda()> > >::operator()(void) (this=0x555556d89af8) at /usr/include/c++/13/bits/std_thread.h:299
#25 0x0000555555e0850c in std::thread::_State_impl<std::thread::_Invoker<std::tuple<facebook::cachelib::navy::ThreadPoolExecutor::ThreadPoolExecutor(uint32_t, folly::StringPiece)::<lambda()> > > >::_M_run(void) (this=0x555556d89af0) at /usr/include/c++/13/bits/std_thread.h:244
#26 0x00007ffff44ecdb4 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#27 0x00007ffff409caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#28 0x00007ffff4129c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

my questions are:

1. is the error above caused by some unsatisfying factor in env like memory error or is some normal log just indicated some event like cache miss?
2. if it's a env issue, is it a must to have 48G+ memory? why small memory related config still need big memory usage?
3. if it's normal issue, is it mean that the test only interact with RAM for some time, how long before it start interact with disk?

[PapperYZ]

Try this and see if it works: #387 (comment)

[pbhandar2]

Closing this discussion as it has been inactive. If this is still relevant, then please create a new discussion post and we will get back to you as soon as we can. Thank you!