close
Skip to content

DynamicConfigManager will throw exception when close #2017

@LiebingYu

Description

@LiebingYu

Search before asking

  • I searched in the issues and found nothing similar.

Fluss version

main (development)

Please describe the bug 🐞

When closing TabletServer, an IllegalStateException will always thrown when closing DynamicConfigManager.

2025-11-25 17:15:31,506 WARN  org.apache.fluss.server.ServerBase                           [] - RECEIVED SIGNAL 2: SIGINT. Shutting down as requested.
2025-11-25 17:15:31,507 INFO  org.apache.fluss.server.tablet.TabletServer                  [] - Shutting down Tablet server (JVM_SHUTDOWN).
2025-11-25 17:15:31,507 INFO  org.apache.fluss.server.tablet.TabletServer                  [] - Starting controlled shutdown.
2025-11-25 17:15:31,532 INFO  org.apache.fluss.rpc.netty.server.RequestProcessorPool       [] - Shutting down Fluss request processor pool.
2025-11-25 17:15:31,533 INFO  org.apache.fluss.server.zk.ZooKeeperClient                   [] - Closing...
2025-11-25 17:15:31,534 INFO  org.apache.fluss.shaded.curator5.org.apache.curator.framework.imps.CuratorFrameworkImpl [] - backgroundOperationsLoop exiting
2025-11-25 17:15:31,646 INFO  org.apache.fluss.shaded.zookeeper3.org.apache.zookeeper.ZooKeeper [] - Session: 0x10054806144002a closed
2025-11-25 17:15:31,646 INFO  org.apache.fluss.shaded.zookeeper3.org.apache.zookeeper.ClientCnxn [] - EventThread shut down for session: 0x10054806144002a
2025-11-25 17:15:33,727 INFO  org.apache.fluss.rpc.netty.client.NettyClient                [] - Netty client was shutdown successfully.
2025-11-25 17:15:33,728 INFO  org.apache.fluss.server.kv.KvManager                         [] - Shutting down KvManager
2025-11-25 17:15:33,728 INFO  org.apache.fluss.server.kv.KvManager                         [] - Shut down KvManager complete.
2025-11-25 17:15:33,728 INFO  org.apache.fluss.server.log.LogManager                       [] - Shutting down LogManager.
2025-11-25 17:15:33,737 INFO  org.apache.fluss.server.log.LogManager                       [] - Shut down LogManager complete.
2025-11-25 17:15:33,737 INFO  org.apache.fluss.server.replica.fetcher.ReplicaFetcherManager [] - Shutting down fetcher manager.
2025-11-25 17:15:33,739 INFO  org.apache.fluss.server.replica.fetcher.ReplicaFetcherManager [] - Shut down fetcher manager Completed.
2025-11-25 17:15:33,740 INFO  org.apache.fluss.server.replica.delay.DelayedOperationManager$ExpiredOperationReaper [] - Shutting down
2025-11-25 17:15:33,740 INFO  org.apache.fluss.server.replica.delay.DelayedOperationManager$ExpiredOperationReaper [] - Stopped
2025-11-25 17:15:33,740 INFO  org.apache.fluss.server.replica.delay.DelayedOperationManager$ExpiredOperationReaper [] - Shutdown completed
2025-11-25 17:15:33,740 INFO  org.apache.fluss.server.replica.delay.DelayedOperationManager$ExpiredOperationReaper [] - Shutting down
2025-11-25 17:15:33,741 INFO  org.apache.fluss.server.replica.delay.DelayedOperationManager$ExpiredOperationReaper [] - Stopped
2025-11-25 17:15:33,741 INFO  org.apache.fluss.server.replica.delay.DelayedOperationManager$ExpiredOperationReaper [] - Shutdown completed
2025-11-25 17:15:33,741 INFO  org.apache.fluss.server.authorizer.ZkNodeChangeNotificationWatcher [] - Stopping TableChangeWatcher
2025-11-25 17:15:33,741 ERROR org.apache.fluss.shaded.curator5.org.apache.curator.framework.imps.CuratorFrameworkImpl [] - Background exception was not retry-able or retry gave up
java.lang.IllegalStateException: Client is not started
	at org.apache.fluss.shaded.curator5.org.apache.curator.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:502) ~[fluss-shaded-zookeeper-3.8.3-1.0-incubating.jar:3.8.3-1.0-incubating]
	at org.apache.fluss.shaded.curator5.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:139) ~[fluss-shaded-zookeeper-3.8.3-1.0-incubating.jar:3.8.3-1.0-incubating]
	at org.apache.fluss.shaded.curator5.org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:1012) ~[fluss-shaded-zookeeper-3.8.3-1.0-incubating.jar:3.8.3-1.0-incubating]
	at org.apache.fluss.shaded.curator5.org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:667) ~[fluss-shaded-zookeeper-3.8.3-1.0-incubating.jar:3.8.3-1.0-incubating]
	at org.apache.fluss.shaded.curator5.org.apache.curator.framework.imps.RemoveWatchesBuilderImpl.pathInBackground(RemoveWatchesBuilderImpl.java:228) ~[fluss-shaded-zookeeper-3.8.3-1.0-incubating.jar:3.8.3-1.0-incubating]
	at org.apache.fluss.shaded.curator5.org.apache.curator.framework.imps.RemoveWatchesBuilderImpl.forPath(RemoveWatchesBuilderImpl.java:196) ~[fluss-shaded-zookeeper-3.8.3-1.0-incubating.jar:3.8.3-1.0-incubating]
	at org.apache.fluss.shaded.curator5.org.apache.curator.framework.imps.RemoveWatchesBuilderImpl.forPath(RemoveWatchesBuilderImpl.java:34) ~[fluss-shaded-zookeeper-3.8.3-1.0-incubating.jar:3.8.3-1.0-incubating]
	at org.apache.fluss.shaded.curator5.org.apache.curator.framework.recipes.watch.PersistentWatcher.close(PersistentWatcher.java:117) ~[fluss-shaded-zookeeper-3.8.3-1.0-incubating.jar:3.8.3-1.0-incubating]
	at org.apache.fluss.shaded.curator5.org.apache.curator.framework.recipes.cache.CuratorCacheImpl.close(CuratorCacheImpl.java:105) ~[fluss-shaded-zookeeper-3.8.3-1.0-incubating.jar:3.8.3-1.0-incubating]
	at org.apache.fluss.server.authorizer.ZkNodeChangeNotificationWatcher.stop(ZkNodeChangeNotificationWatcher.java:95) ~[classes/:?]
	at org.apache.fluss.server.DynamicConfigManager.close(DynamicConfigManager.java:80) ~[classes/:?]
	at org.apache.fluss.server.tablet.TabletServer.stopServices(TabletServer.java:427) ~[classes/:?]
	at org.apache.fluss.server.tablet.TabletServer.closeAsync(TabletServer.java:290) ~[classes/:?]
	at org.apache.fluss.server.ServerBase.lambda$addShutDownHook$0(ServerBase.java:152) ~[classes/:?]
	at org.apache.fluss.server.utils.ShutdownHookUtil.lambda$addShutdownHook$0(ShutdownHookUtil.java:42) ~[classes/:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
2025-11-25 17:15:33,747 INFO  org.apache.fluss.server.ServerBase                           [] - Terminating TabletServer process with exit code 1.

That because ZookeeperClient close first. We should close DynamicConfigManager before ZookeeperClient.

Solution

No response

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No fields configured for Bug.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions