v1.28.0

Dragonfly v1.28.0 - Spring Edition 🌵

This marks our second release for the year, with lots of bugs closed and probably some new bugs introduced.

Important fixes and changes:

1. Deadlocked transaction bugs #4590 #4647 #4685
2. Sharded pubsub (API-7) : #4702 #4518 #4529
3. Fix Streams replication #4591 #4737
4. Numerous cluster improvements #4727 #4786 #4767

What's Changed

• fix(search_family): Add options test for the FT.AGGREGATE command by @BagritsevichStepan in #4479
• fix(stream_family): Fix replication for the XADD and XTRIM commands by @BagritsevichStepan in #4591
• chore(search_family): Simplify FT.SEARCH reply code by @BagritsevichStepan in #4553
• feat(zset family): Implement ZDIFFSTORE by @mkaruza in #4582
• fix(seeder): Support stream types in the Seeder by @BagritsevichStepan in #4574
• fix: debian path in dragonfly.service by @romange in #4594
• fix: tls replication without ca verification by @kostasrim in #4539
• fix(server): list move on single shard wake blocking transaction by @adiholden in #4590
• fix: race when bumping items while loading a snapshot by @kostasrim in #4564
• fix: multi/exec CROSSSLOT error processing by @BorysTheDev in #4598
• test: added test_keys_expiration_during_migration by @BorysTheDev in #4558
• chore: fix test_snapshot by @kostasrim in #4607
• chore: fix glob translation with \ at the end by @romange in #4608
• feat: support lz4 compression method by @romange in #4610
• fix: test_keys_expiration_during_migration by @BorysTheDev in #4613
• chore: add zstd_test by @romange in #4614
• chore: simplify transaction code by @romange in #4602
• fix: epoll reg tests running on iouring by @kostasrim in #4612
• chore: use macos-14 for the daily macos build by @kostasrim in #4609
• chore: introduce docker release pipeline by @romange in #4618
• chore: connection pipeline cache does not shrink by @kostasrim in #4491
• chore: implement sharded pub/sub by @kostasrim in #4518
• chore: introduce background deletions of DenseSet objects by @romange in #4496
• fix: revert reflex based matching due to performance problems by @romange in #4626
• chore: allow yielding in Scan command by @kostasrim in #4624
• refactor: simplify journal and restore streamer cancellation by @BorysTheDev in #4549
• server: rdb loader big string loading in chunks by @adiholden in #4623
• chore: remove redundant calls to GetSlice by @romange in #4635
• server: redis parser reject big bulk read by @adiholden in #4627
• chore: Improve loading of sorted sets by @romange in #4631
• fix: invalid BUSYKEY response by @romange in #4637
• chore: propagate reserve data upon restore by @romange in #4641
• chore: get rid of sds in SortedMap::AddElem by @romange in #4638
• fix: improve test_send_delay_metric test by @romange in #4642
• fix: do not flush replicaof if failed to connect by @kostasrim in #4636
• fix: performance degradation with Reflex matching by @romange in #4634
• chore: get rid of lispack stats by @romange in #4644
• chore: add listpack statistics to objhist by @romange in #4650
• chore: allow sampling of topk hottest keys. by @romange in #4649
• feat: implement unique keys counting using hll by @romange in #4652
• fix: pull latest helio by @romange in #4651
• chore: unsubscribe pub/sub connections after cluster migration by @kostasrim in #4529
• fix: bring back the helm step to docker release action by @romange in #4657
• fix: do not crash upon inconsistent expire iterator by @romange in #4658
• chore: replace try assert false with pytest.raises by @kostasrim in #4662
• server: disable single shard tx optimization on scheduling by @adiholden in #4647
• fix: race condition in ReplicaOfInternal by @kostasrim in #4653
• Support Cluster configuration for dfly_bench by @romange in #4664
• CI: add slow regression tests by @adiholden in #4646
• fix: replication error message and scan command yield by @kostasrim in #4665
• feat: add migration error statistic by @BorysTheDev in #4643
• chore: refactor ServerFamily::Info by @romange in #4671
• fix(search_family): Add option tests for the FT.CREATE command by @BagritsevichStepan in #4678
• chore: introduce DEBUG COMPRESSION by @romange in #4620
• chore: cluster related clean ups by @romange in #4683
• fix: SendInvalidationTrackingMessage should not block. by @romange in #4680
• fix: reduce stack usage on Fedora by @romange in #4690
• fix(server): deadlock with replicaof inside multi by @adiholden in #4685
• fix(metrics): add corresponding comments to command metrics by @Yomanz in #4674
• feat: Added a flag to ignore key expiry. by @H4R5H1T-007 in #4667
• feat(cluster_mgr): Add argument to set path to dragonfly binary by @mkaruza in #4695
• fix: possible preemption under FiberAtomicGuard by @romange in #4692
• chore: add debug printings to SetExisting by @romange in #4694
• chore: Add debug logs to help tracking transactional deadlocks by @romange in #4669
• chore: snapshot/tiering improvements by @romange in #4661
• chore: disable failing epoll tests by @kostasrim in #4699
• feat: client unpause and client help by @kostasrim in #4630
• fix: ZMSCORE return value if key does not exist by @vyavdoshenko in #4697
• fix: prevent tx_executor reading from sock if operation is canceled by @BorysTheDev in #4704
• feat(set_family): Update TTL for existing fields in SADDEX by @abhijat in #4700
• chore: update CONTRIBUTING.md by @kostasrim in #4703
• chore: minor clean ups before introducing ProvidedBuffers by @romange in #4709
• refactor: clean cluster code by @BorysTheDev in #4707
• fix: Remove the dfly logo print from pytest run #4682 by @vyavdoshenko in #4706
• fix(json_family): Fix JSON.SET handling for nested fields by @BagritsevichStepan in #4710
• fix(dfly_bench): support dns resolution for cluster hosts by @romange in #4715
• fix: buffer overrun in GetRandomHex by @romange in #4717
• feat(set_family): Add support for KEEPTTL to SAddEx by @abhijat in #4712
• fix: preemption in atomic section of heartbeat by @kostasrim in #4720
• Revert "chore: minor clean ups before introducing ProvidedBuffers (#4… by @adiholden in #4719
• feat(server): Add PUBSUB SHARDCHANNELS/SHARDNUMSUB by @mkaruza in #4702
• fix(dfly_bench): track hit rate for mget command by @romange in https://github.com/dragonflydb/dr...

v1.27.3

This is a patch release.

What's Changed

• fix: invalid reference inside GetQueueBackpressure (#4789)
• fix(cluster) : moved error port number on migration finish (#4776)

Full Changelog: v1.27.2...v1.27.3

v1.27.2

This is a patch release.

What's Changed

• fix: add corresponding comments to command metrics (#4674)
• fix: deadlock with replicaof inside multi (#4685)
• feat: allow cluster READONLY command (#4727)
• fix: disable single shard tx optimization on scheduling (#4647)
• fix: preemption in atomic section of heartbeat (#4720)

Full Changelog: v1.27.1...v1.27.2

v1.27.1

This is a patch release.

What's Changed

• fix: wrong binary path in deb package (#4593)

Update urgency: low
Full Changelog: v1.27.0...v1.27.1

v1.27.0

Dragonfly v1.27.0 - a Valentine Edition 💘

This marks our first release for the year, featuring numerous stability fixes and performance enhancements.

Important features and changes:

1. Fixes around JSON+SEARCH: #4232 #4314
2. Support for memcached meta protocol #4362 #4398 #4366
3. Fixes in tiering code #4395 #4388 #4416
4. Added LMPOP #4504 and ZMPOP #4385

What's Changed

• fix: test_network_disconnect_during_migration by @BorysTheDev in #4345
• refactor(rdb_saver): Add SnapshotDataConsumer to SliceSnapshot by @BagritsevichStepan in #4287
• chore(rax_tree): Introduce raxFreeWithCallback call in RaxTreeMap destructor by @BagritsevichStepan in #4255
• refactor(search_family): Add Aggregator class by @BagritsevichStepan in #4290
• fix: avoid on stack allocation of lz4 compression context by @romange in #4322
• chore: update helio and improve our stack overflow resiliency by @romange in #4349
• fix: configure jsonconst double parser to use std::from_chars by @romange in #4360
• fix(search_family): Support boolean and nullable types in indexes by @BagritsevichStepan in #4314
• fix(search_family): Support multiple fields in SORTBY option in the FT.AGGREGATE command. SECOND PR by @BagritsevichStepan in #4232
• test: skip test_network_disconnect_during_migration by @BorysTheDev in #4359
• fix(json_family): Remove std::endl during logging in ParseJsonPath method by @BagritsevichStepan in #4363
• feat: add support for meta memcache commands by @romange in #4362
• chore(test): disable failing test untill fixed by @adiholden in #4367
• chore(deps): bump the actions group across 1 directory with 2 updates by @dependabot in #4361
• feat: support memcache meta responses by @romange in #4366
• fix(pytest): fix failing test test_replication_timeout_on_full_sync by @adiholden in #4370
• chore(json_family): Remove redundant JSON value copying by @BagritsevichStepan in #4375
• feat(rdb_saver): Support big value serialization for stream by @BagritsevichStepan in #4376
• chore(snapshot): Small cleanup in Snapshot code by @BagritsevichStepan in #4377
• fix: stream memory counting durin snapshot loading by @BorysTheDev in #4346
• fix: return value for DflyMigrateAck by @BorysTheDev in #4379
• chore(server): cleanup replication shard sync execution by @adiholden in #4374
• test: fix test_network_disconnect_during_migration by @BorysTheDev in #4378
• fix(snapshot_test): Fix test_big_value_serialization_memory_limit after adding streams support by @BagritsevichStepan in #4383
• fix: ExternalAllocator::Free with large sizes by @romange in #4388
• fix(server): debug populate consume less memory by @adiholden in #4384
• refactor: slot_set don't use stack memory anymore by @BorysTheDev in #4386
• feat:Adding support for ZMPOP command by @guyzilla in #4385
• fix(replication): do not log to journal on callback fail by @adiholden in #4392
• chore: minor refactorings around dense_set deletions by @romange in #4390
• feat(server): refactor allow preempt on journal record by @adiholden in #4393
• fix: allow cluster node load snapshot bigger than maxmemory by @BorysTheDev in #4394
• fix: properly clear tiered stashes upon expirations by @romange in #4395
• feat: support deletions with meta protocol by @romange in #4398
• fix(tests): check cluster big snapshot in unit test by @adiholden in #4403
• chore(pytest): add timeout per test by @adiholden in #4404
• fix: provide resp3 option to CapturingReplyBuilder by @BorysTheDev in #4400
• chore: deprecate unneeded runtime flags by @romange in #4405
• feat: Yield inside huge values migration serialization by @chakaz in #4197
• chore(pytest): add check for rss grow in replicaiton big values by @adiholden in #4406
• feat: allow querying of json objects stored as strings by @romange in #4399
• test: move ReplicaofRejectOnLoad test from pytest into unit tests by @BorysTheDev in #4410
• chore: Increase migration wait time in test_cluster_migration_huge_container by @chakaz in #4409
• chore: Introduce small buffer in redis parser by @romange in #4076
• fix: invalid iterator case in Qlist::Erase by @romange in #4414
• test: add test migration in cache mode by @BorysTheDev in #4413
• fix: ExtentTree use of invalidated iterator by @kostasrim in #4416
• chore: fix build on FreeBSD by @chakaz in #4412
• chore: reduce number of logs in cluster tests by @BorysTheDev in #4419
• fix: geosearch and georadius response format by @kostasrim in #4420
• chore: split geo and zset families by @kostasrim in #4421
• fix: switch to SHUTTING_DOWN state unconditionally by @romange in #4408
• chore: split zset and geo family tests by @kostasrim in #4425
• refactor: remove transaction lib on cluster code dependency by @BorysTheDev in #4417
• chore: Increase migration wait time in test_cluster_migration_huge_container by @chakaz in #4427
• chore: update jsoncons version to 0.178 by @romange in #4368
• fix: improve error propagation with RESTORE commands by @romange in #4428
• fix: show managed stuff on privileged port by @romange in #4433
• chore: dragonfly connection refactorings by @romange in #4434
• fix: reply_builder should properly serialize bulk strings endings. by @romange in #4441
• chore: pull latest helio by @romange in #4446
• chore: Remove template Callbacks by @BagritsevichStepan in #4431
• fix: resp inline parsing correctly resets itself by @romange in #4458
• fix(ci): sanitizers build by @kostasrim in #4457
• chore: Add stats print for slot migrations by @chakaz in #4456
• fix: slot calculation during transaction squashing by @BorysTheDev in #4460
• fix(sanitizers): Set stack size to 64 KB during tests for Sanitizers by @BagritsevichStepan in #4454
• Fix test hypothesis by @cunla in #4380
• feat(dev): allow monitoring a valkey server on localhost by @romange in #4467
• chore: add Dash::Prefetch function by @romange in #4476
• chore: dfly_bench now accepts data and score template names by @romange in #4477
• chore: replace deprecated HELIO_STACK_CHECK by @kostasrim in #4465
• chore: Fix all clang build warnings by @chakaz in #4475
• fix: hiredis requires df to report version >7.4 by @kostasrim in #4474
• chore: make per-thread QueueBackpressure objects global by @romange in #4482
• chore: OpManagerTest fixes by @romange in #4480
• chore: run replication reg tests with epoll by @kostasrim in #4426
• chore: remove pubsub semantics from Connection::WeakRef by @romange in https://github.com/dragonflydb/dragonfl...

v1.26.4

This is a patch release.

What's Changed

• fix: huge TTL in RESTORE cmd is rounded down to kMaxExpireDeadlineMs (#4589)
• fix: fix: incorrect passing of db_index in EVAL transactions (#4586)

Update urgency: low

Full Changelog: v1.26.3...v1.26.4

v1.26.3

This is a patch release.

What's Changed

• fix(search_family): Cherry pick commit "Support boolean and nullable types in indexes" (#4560)
• fix: a rare deadlock with blmove and multi-db (#4568)
• fix server: fix deadlock blmove does not conclude on error (#4561)
• fix: Bug GEQ range query bug (#4557) - fixes the issue that mastodon users experienced with Dragonfly

Due to #4557 it is strongly advised to update your dragonfly server to v1.26.3 if you use sorted sets / ZRANGEBYSCORE queries.

Full Changelog: v1.26.2...v1.26.3

v1.25.7

This is a patch release.

What's Changed

• fix: a rare deadlock with blmove and multi-db (#4568)
• fix: Bug GEQ range query bug (#4557) - fixes the issue that mastodon users experienced with Dragonfly

Due to #4557 it is strongly advised to update your dragonfly server to either v1.25.7 or v1.26.3
if you use sorted sets and ZRANGEBYSCORE queries.

v1.26.2

This is a patch release.

What's Changed

• fix: add Lua force atomicity flag (#4523)
  Fixes an issue with sidekiq-pro framework.
• fix: Do not bump elements during RDB load (#4507)
• fix: crash for delconsumer during stream reading (#4513)

Full Changelog: v1.26.1...v1.26.2

v1.26.1

This is a patch release.

What's Changed

• fix: data race in LREM operation (#4414)
• fix: admin port should show all the info data (#4433)
• fix: resp3 was ignored under squashing, zrange + multi did was inconsistent (#4400)
• fix: data tiering fixes: #4395, #4388 #4416
• fix: bulk string ending were not properly rendered (#4441)

Full Changelog: v1.26.0...v1.26.1