Ledger DB panic when read range query

[dviejokfs]

Sometimes we are getting this panic error on the peers:

panic: runtime error: slice bounds out of range [370:369]

goroutine 787794892 [running]:
github.com/syndtr/goleveldb/leveldb.tFiles.newIndexIterator({0xc00a752000?, 0x2f5e?, 0x2f5e}, 0xc00107d320, 0xc00100bed0, 0xc004aacfc0, 0xc000fffe20)
	/vendor/github.com/syndtr/goleveldb/leveldb/table.go:308 +0x325
github.com/syndtr/goleveldb/leveldb.(*version).getIterators(0xc015845b00, 0xabcfc8?, 0xc000fffe20)
	/vendor/github.com/syndtr/goleveldb/leveldb/version.go:269 +0x135
github.com/syndtr/goleveldb/leveldb.(*DB).newRawIterator(0xc000418700, 0x0, {0x0, 0x0, 0x1?}, 0xc004aacfc0, 0xc000fffe20)
	/vendor/github.com/syndtr/goleveldb/leveldb/db_iter.go:41 +0xfb
github.com/syndtr/goleveldb/leveldb.(*DB).newIterator(0xc000418700, 0xc000101000?, {0x0?, 0x0?, 0x0?}, 0x1660c9e1, 0xc00dcd7850?, 0xc000fffe20)
	/vendor/github.com/syndtr/goleveldb/leveldb/db_iter.go:79 +0x2db
github.com/syndtr/goleveldb/leveldb.(*DB).NewIterator(0xc000418700, 0x18?, 0x18a5cc0?)
	/vendor/github.com/syndtr/goleveldb/leveldb/db.go:897 +0x134
github.com/hyperledger/fabric/common/ledger/util/leveldbhelper.(*DB).GetIterator(0x0?, {0xc005968be0?, 0x0?, 0xc000174088?}, {0xc00e91bd60?, 0x1b58927?, 0x28?})
	/common/ledger/util/leveldbhelper/leveldb_helper.go:159 +0x137
github.com/hyperledger/fabric/common/ledger/util/leveldbhelper.(*DBHandle).GetIterator(0xc003f90520, {0xc005256bd0, 0x8d, 0xc00dcd79f0?}, {0xc014d95fc0, 0x3a, 0x200000003?})
	/common/ledger/util/leveldbhelper/leveldb_provider.go:297 +0x371
github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/stateleveldb.(*versionedDB).GetStateRangeScanIteratorWithPagination(0xc003c06810, {0xc015015ea0, 0xb}, {0xc00c0d7d80, 0x80}, {0xc00e77b6b0, 0x2d}, 0x3e8)
	/core/ledger/kvledger/txmgmt/statedb/stateleveldb/stateleveldb.go:167 +0x478
github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/txmgr.newResultsItr({0xc015015ea0, 0xb}, {0xc00c0d7d80, 0x80}, {0xc00e77b6b0, 0x2d}, 0x5?, {0x1d351a8?, 0xc003c068a0?}, 0xc00c699e40, ...)
	/core/ledger/kvledger/txmgmt/txmgr/query_executor.go:410 +0x92
github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/txmgr.(*queryExecutor).GetStateRangeScanIteratorWithPagination(0xc0145a71f0, {0xc015015ea0?, 0x1869b80?}, {0xc00c0d7d80?, 0xc00dcd7c88?}, {0xc00e77b6b0?, 0xc00784eb60?}, 0x10?)
	/core/ledger/kvledger/txmgmt/txmgr/query_executor.go:151 +0x15f
github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/txmgr.(*txSimulator).GetStateRangeScanIteratorWithPagination(0xc005256b40?, {0xc015015ea0?, 0x90?}, {0xc00c0d7d80?, 0xc014193dc0?}, {0xc00e77b6b0?, 0xab292c?}, 0x19dbca0?)
	/core/ledger/kvledger/txmgmt/txmgr/tx_simulator.go:183 +0x10e
github.com/hyperledger/fabric/core/chaincode.(*Handler).HandleGetStateByRange(0xc00c3738c0, 0xc012139d80, 0xc0048940a0)
	/core/chaincode/handler.go:756 +0x296
github.com/hyperledger/fabric/core/chaincode.(*Handler).HandleTransaction(0xc00c3738c0, 0xc012139d80, 0xc012ea4d30)
	/core/chaincode/handler.go:253 +0x373
created by github.com/hyperledger/fabric/core/chaincode.(*Handler).handleMessageReadyState in goroutine 671114636
	/core/chaincode/handler.go:195 +0x3e9

It happens when there's high concurrency and special read range query is executed.

At the time of writing this discussion, I don't have more details about the query since we would need to change fabric to add logging for this specific issue and it's almost impossible for us to reproduce.

On the other hand, I think that Fabric shouldn't panic when there's an error on a transaction execution, it should log an error, but not kill the entire peer since it can be exploited as a Denial of Service.

[pfi79]

What version are you running this on?
I have recently updated goleveldb to the latest version.
Maybe the problem has gone away, but not sure.

[KoushikGavini]

Hi @pfi79, we see this error in both 2.5.5 and 3.0.0. Here are the relevant PRs,

#5078
#5001

[pfi79]

I updated the package after the release of v3.0.0. So it's on top of main now.
What can be done:

1. You will take build fabric from main and check if this error appears there.
2. You wait for the new version to be released and check it. And we come back to this question.

[dviejokfs]

@pfi79
I see the upgrade, it went from v1.0.1-0.20210305035536-64b5b1c73954 and v1.0.1-0.20220721030215-126854af5e6d

These are the changes @pfi79 https://github.com/syndtr/goleveldb/compare/64b5b1c73954..126854af5e6d

I'm honestly not sure if that would fix it, right now we are not getting these panic errors.