close
Skip to content

Support slot-based data migration#430

Merged
ShooterIT merged 20 commits into
apache:unstablefrom
ChrisZMF:unstable
Jan 27, 2022
Merged

Support slot-based data migration#430
ShooterIT merged 20 commits into
apache:unstablefrom
ChrisZMF:unstable

Conversation

@ChrisZMF
Copy link
Copy Markdown
Contributor

@ChrisZMF ChrisZMF commented Dec 17, 2021

resolves #412

Support Commands

  • CLUSTERX MIGRATE $slot $dst_nodeid
    $dst_nodeid is the node id of destination server in the cluster. See [NEW] Support redis cluster mode [NEW] Support redis cluster mode #219 for more details.
  • CLUSTER IMPORT $slot $state
    It is an internal command which will be sent by the source server to notify the destination server to prepare for data importing. This command cannot be used by clients.
  • Migrating status and importing status are merged into the output of CLUSTER INFO.
    Example:
    Source node info:
127.0.0.1:30001> cluster info
cluster_state:ok
cluster_slots_assigned:16384
......
migrating_slot: 0
destination_node: kvrockskvrockskvrockskvrockskvrocksnode3
migrating_state: success

Destination node info:

127.0.0.1:30003> cluster info
cluster_state:ok
cluster_slots_assigned:16384
......
importing_slot: 0
import_state: success

After migration slot, you should use CLUSTERX SETSLOT #463 to change cluster slot distribution.

 - code optimization
@ShooterIT
Copy link
Copy Markdown
Member

@ChrisZMF Please resolve conflicts (related to kvrocks2redis)

Copy link
Copy Markdown
Member

@ShooterIT ShooterIT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing work, let's review ASAP @git-hulk

Comment thread src/batch_parser.h
Comment thread src/batch_parser.h
Comment thread src/batch_parser.h Outdated
Comment thread src/batch_parser.cc
Comment thread src/batch_parser.cc
Comment thread src/redis_db.cc Outdated
Comment thread src/redis_metadata.cc
Comment thread src/redis_request.h Outdated
Comment thread src/server.h Outdated
Comment thread src/slot_import.h Outdated
Comment thread src/cluster.cc Outdated
Comment thread src/cluster.cc Outdated
Comment thread src/cluster.cc Outdated
Comment thread src/cluster.cc Outdated
Comment thread src/cluster.cc Outdated
Comment thread src/cluster.cc Outdated
Comment thread src/config.cc
Comment thread src/config.cc
Comment thread src/redis_cmd.cc Outdated
Comment thread src/redis_db.cc Outdated
@ShooterIT
Copy link
Copy Markdown
Member

Hi @ChrisZMF I have merge CLUSTER SETSLOT PR, please resolve the conflicts and implement what we have reached an agreement!

Comment thread src/config.cc Outdated
Comment thread src/redis_cmd.cc Outdated
Comment thread src/server.cc
Comment thread src/server.h Outdated
Comment thread src/util.cc Outdated
Comment thread src/slot_migrate.cc Outdated
Comment thread src/slot_migrate.cc Outdated
Comment thread src/slot_migrate.cc Outdated
Comment thread src/slot_migrate.cc Outdated
Comment thread src/batch_extractor.h Outdated
Comment thread src/cluster.cc Outdated
Comment thread src/server.cc Outdated
Comment thread src/server.cc Outdated
Comment thread src/batch_extractor.cc
Copy link
Copy Markdown
Member

@ShooterIT ShooterIT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, Cheers!
@ChrisZMF Thanks for your great PR which authentically makes kvrocks scalable , also thanks for your patience of keeping following CRs and tolerating my preferences.

@git-hulk
Copy link
Copy Markdown
Member

Cheers! @ShooterIT I didn't have other suggestions, can you help to summary this PR.

@ChrisZMF
Copy link
Copy Markdown
Contributor Author

I am really glad this PR is accepted. Thank you for the suggestions you given for this PR to make it better. It really makes a lot of sense to me. @ShooterIT @git-hulk

@ShooterIT ShooterIT merged commit 805dc84 into apache:unstable Jan 27, 2022
ShooterIT pushed a commit to ShooterIT/kvrocks that referenced this pull request Jan 27, 2022
A new command CLUSTERX MIGRATE is used for migrate slot data, slot-based migration
process mainly includes the following stages: migrating existing data and migrating
incremental data.

Command format:
CLUSTERX MIGRATE $slot $dst_nodeid
  - $slot is the slot which is to migrate
  - $dst_nodeid is the node id of destination server in the cluster.

We also introduce an internal command CLUSTER IMPORT for importing the migrating
slot data into destination server.

Migration status are shown into the output of CLUSTER INFO command.

After migration slot, you also should use CLUSTERX SETSLOT command to change cluster slot
distribution.

For more details, please see apache#412 and apache#430
@ShooterIT ShooterIT mentioned this pull request Jan 27, 2022
ShooterIT pushed a commit to ShooterIT/kvrocks that referenced this pull request Jan 28, 2022
A new command CLUSTERX MIGRATE is used for migrate slot data, slot-based migration
process mainly includes the following stages: migrating existing data and migrating
incremental data.

Command format:
CLUSTERX MIGRATE $slot $dst_nodeid
  - $slot is the slot which is to migrate
  - $dst_nodeid is the node id of destination server in the cluster.

We also introduce an internal command CLUSTER IMPORT for importing the migrating
slot data into destination server.

Migration status are shown into the output of CLUSTER INFO command.

After migration slot, you also should use CLUSTERX SETSLOT command to change cluster slot
distribution.

For more details, please see apache#412 and apache#430
ShooterIT pushed a commit that referenced this pull request Jan 28, 2022
A new command CLUSTERX MIGRATE is used for migrate slot data, slot-based migration
process mainly includes the following stages: migrating existing data and migrating
incremental data.

Command format:
CLUSTERX MIGRATE $slot $dst_nodeid
  - $slot is the slot which is to migrate
  - $dst_nodeid is the node id of destination server in the cluster.

We also introduce an internal command CLUSTER IMPORT for importing the migrating
slot data into destination server.

Migration status are shown into the output of CLUSTER INFO command.

After migration slot, you also should use CLUSTERX SETSLOT command to change cluster slot
distribution.

For more details, please see #412 and #430
Comment thread src/redis_cmd.cc
Comment on lines +3470 to +3473
if (svr->GetConfig()->cluster_enabled) {
svr->slot_migrate_->SetMigrateStopFlag(false);
LOG(INFO) << "Change server role to master, restart migration task";
}
Copy link
Copy Markdown
Member

@enjoy-binbin enjoy-binbin Sep 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn’t this block of code are dead code? since in the beginning we will throw an error if we are in the cluster mode

  Status Execute(Server *svr, Connection *conn, std::string *output) override {
    if (svr->GetConfig()->cluster_enabled) {
      return Status(Status::RedisExecErr, "can't change to slave in cluster mode");
    }

sorry for commenting in this very old PR... i tracked the changes, afraid i am missing something.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[NEW] Support slot-based data migration

4 participants