Commands.txt

cd to bin dir of kafka folder


-------------starting zookeeper server -------------------------------------------------------------------

 ./zookeeper-server-start.sh ../config/zookeeper.properties 

---------------------------------------------------------

before starting the kafka server, edit the server.properties in the config folder.

uncommnent the advertise.listner line of file.

change it to localhost so that zookeeper can understand url to pick the messgaes from.
 
-------------starting Kafka server -----------------------------------------------------------------------

./kafka-server-start.sh ../config/server.properties


------- creating a topic ---------------------------------------------------------------------------------

./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

------- Describe a topic ---------------------------------------------------------------------------------

./kafka-topics.sh --describe --zookeeper localhost:2181 --topic test


----list of topics----------------------------------------------------------------------------------------
./kafka-topics.sh --list --zookeeper localhost:2181

---creating a producer -----------------------------------------------------------------------------------
./kafka-console-producer.sh --broker-list localhost:9092 --topic test


--- creating a consumer ----------------------------------------------------------------------------------

./kafka-console-consumer.sh --zookeeper localhost:9092 --topic test --from-beginning


./kafka-console-consumer.sh --zookeeper localhost:9092 --topic test


----------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------

CREATE TABLE `emp_replicate` (
  `empno` decimal(4,0) NOT NULL,
  `ename` varchar(10) DEFAULT NULL,
  `job` varchar(9) DEFAULT NULL,
  `mgr` decimal(4,0) DEFAULT NULL,
  `hiredate` date DEFAULT NULL,
  `sal` decimal(7,2) DEFAULT NULL,
  `comm` decimal(7,2) DEFAULT NULL,
  `deptno` decimal(2,0) DEFAULT NULL
)

insert into emp-replicate values(7654,'MARTIN','SALESMAN',7698,'1981-09-28',1250.0,1400.0,20);

insert into emp-replicate(empno,ename,job,mgr,hiredate,sal,com,deptno) values(7654,'MARTIN','SALESMAN',7698,'1981-09-28',1250.0,1400.0,20);


CREATE TABLE `emp1` (
  `empno` decimal(4,0) NOT NULL,
  `ename` varchar(10) DEFAULT NULL,
  `job` varchar(9) DEFAULT NULL,
  `mgr` decimal(4,0) DEFAULT NULL,
  `hiredate` date DEFAULT NULL,
  `sal` decimal(7,2) DEFAULT NULL,
  `comm` decimal(7,2) DEFAULT NULL,
  `deptno` decimal(2,0) DEFAULT NULL
)

insert into emp1 values(7654,'MARTIN','SALESMAN',7698,'1981-09-28',1250.0,1400.0,20);


----------------------------------------------------------------------------------------------------------

------------creating mutliple brokers --------------

copy the server.properties file from the config folder and paste it with same name and add extn-1,-2, ...... as required
open these files and edit the broker.id which should be unique
thn edit the logs file location with entended -1,-2 .... to create a new dir for the logs generated by new brokers
uncommnet the listner and advertise listerner

----------------- create a topic with replication and partitions-------------

./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic


./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 3 --topic my-replicated-topic
----- to describe a topic with replications ----
./kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic

------producer for multiple brokers----(jst like regular one but we have created a fault tolerant one by use of 3 servers)

./kafka-console-producer.sh --broker-list localhost:9092 --topic my-replicated-topic


----consumer for mutliple brokers-----
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic


---- testing fault tolerance ---

> ps aux | grep server-1.properties
7564 ttys002    0:15.91 /System/Library/Frameworks/JavaVM.framework/Versions/1.8/Home/bin/java...
> kill -9 7564

killed the lead leader of our broker cluster

chekcking the new leader


./kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic


But the messages are still available for consumption even though the leader that took the writes originally is down:
prashanth@prashanth-VirtualBox:/usr/local/kafka_2.12-1.0.0/bin$ ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic
Topic:my-replicated-topic	PartitionCount:1	ReplicationFactor:3	Configs:
	Topic: my-replicated-topic	Partition: 0	Leader: 0	Replicas: 0,1,2	Isr: 0,2
prashanth@prashanth-VirtualBox:/usr/local/kafka_2.12-1.0.0/bin$ ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
welcome to fault tolerant mutliple broker msg system
now lets see how will the ft works
lets kill a broker
lets kill the leader broker of sys
thats the o server


----list of topics---
./kafka-topics.sh --list --zookeeper localhost:2181

--- java producer's consumer --------- use bootstrap-server as name. since its has updated --
prashanth@prashanth-VirtualBox:/usr/local/kafka_2.12-1.0.0/bin$ ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic java-topic --from-beginning

---- altering a topics config---

./kafka-topics.sh --zookeeper localhost:2181 --alter --topic java-topic --partitions 4