Deploy apache pulsar with external zookeeper

[spesalvi]

Hi All,

I have a zookeeper instance configured in a different namespace. I want to deploy the apache pulsar and use the zookeeper instance that I have in the other namespace.

In the values.yaml file i see an option to disable zookeeper. When I disable that, zookeeper instance is not created.

I also provided externalZookeeperServerList with my external zookeeper endpoints.

Now I notice that externalZookeeperServerList is used when zookeeper component is enabled. So, my zookeeper endpoints are not use by the helm chart and bookie and broker are always trying to connect to 127.0.0.1:2181.

Is there a way to configure pulsar to use external zookeeper instead.

Thanks,
Robert

[lhotari]

@spesalvi Yes, that's possible. Just don't specify externalZookeeperServerList since there's a different purpose for that setting. Instead, use userProvidedZookeepers.

pulsar-helm-chart/charts/pulsar/values.yaml

Lines 898 to 901 in afca5aa

	## optional, you can provide your own zookeeper metadata store for other components
	# to use this, you should explicit set components.zookeeper to false
	#
	# userProvidedZookeepers: "zk01.example.com:2181,zk02.example.com:2181"

[lhotari]

Reopening this since there seems to be changed which cause userProvidedZookeepers not to work as expected.

[lhotari]

It looks like the gap is about initializing the cluster metadata automatically. That only happens if you deploy the Zookeeper component. You will need to initialize the cluster metadata manually. Essentially handling this command manually:

pulsar-helm-chart/charts/pulsar/templates/pulsar-cluster-initialize.yaml

Lines 123 to 139 in afca5aa

	{{- include "pulsar.toolset.zookeeper.tls.settings" . | nindent 12 }}
	export PULSAR_MEM="-Xmx128M";
	bin/pulsar initialize-cluster-metadata \
	--cluster {{ template "pulsar.cluster.name" . }} \
	--zookeeper {{ template "pulsar.zookeeper.connect" . }}{{ .Values.metadataPrefix }} \
	{{- if .Values.pulsar_metadata.configurationStore }}
	--configuration-store {{ template "pulsar.configurationStore.connect" . }}{{ .Values.pulsar_metadata.configurationStoreMetadataPrefix }} \
	{{- else }}
	--configuration-store {{ template "pulsar.zookeeper.connect" . }}{{ .Values.metadataPrefix }} \
	{{- end }}
	--web-service-url http://{{ template "pulsar.fullname" . }}-{{ .Values.broker.component }}.{{ template "pulsar.namespace" . }}.svc.{{ .Values.clusterDomain }}:{{ .Values.broker.ports.http }}/ \
	--web-service-url-tls https://{{ template "pulsar.fullname" . }}-{{ .Values.broker.component }}.{{ template "pulsar.namespace" . }}.svc.{{ .Values.clusterDomain }}:{{ .Values.broker.ports.https }}/ \
	--broker-service-url pulsar://{{ template "pulsar.fullname" . }}-{{ .Values.broker.component }}.{{ template "pulsar.namespace" . }}.svc.{{ .Values.clusterDomain }}:{{ .Values.broker.ports.pulsar }}/ \
	--broker-service-url-tls pulsar+ssl://{{ template "pulsar.fullname" . }}-{{ .Values.broker.component }}.{{ template "pulsar.namespace" . }}.svc.{{ .Values.clusterDomain }}:{{ .Values.broker.ports.pulsarssl }}/ ;
	{{- if .Values.extraInitCommand }}
	{{ .Values.extraInitCommand }}
	{{- end }}

. by running ./bin/pulsar initialize-cluster-metadata from a machine that can access the target Zookeeper.