We know that MQTT can be and is widely used for Internet of Things Applications. But do you know that mqtt can be a good choice of sending data in local networks too? Instead of juggling with TCP or HTTP request and running webserver, mqtt can be a super simple, yet reliable solution for local network data exchange between raspberry pi to raspberry pi or Raspberry pi to computer. You can use mqtt for any small requirement of sending data from one computer to another computer. or you can use mqtt to send data to PC from raspberry pi. You can also use mqtt to send data to raspeberry pi from PC. If you want to know how to setup raspberry pi for the first time, you can read it here
In order to do any of these things, first thing is to understand how mqtt works? and what are the advantages over the conventional client-server communication
How Client-server communication Works?
In a very conventional sense, whenever we talk about 2 computers talking to each other, the general way of communication is Client -> Server Communication.
Lets say when 2 computers wants to talk to each other, unless we’re using a direct TCP communicaiton, we often need server. The typical configuration will look like this
Now in above structure, what usually happens is when client 1 sends some data, the server has to save it. And when client 2 sends a request for that data then server will serve it. Lets take an example of IoT project and client 1 is sending temperature and client 2 wants to have that temperature. The communication goes like this
- Client 1 reads temperature and sends it to Server
- Server receives it and stores it somehow
- Client 2 requests server about data
- Server responds with the available data
- client 2 gets requested data
How MQTT Works?
Now if the same communication is to be done using mqtt, lets see how it works.
First, in MQTT, there is no need to save the data at the server end. The data saving into any database on server is completely optional. Instead the server immediately sends that data back to the clients who need it by fetching from its own cache. Now how to know which client wants that data? In order to know which client wants which data and which client sends which data, mqtt uses something called topics. A topic is basically a virtual communication ID and is better understood in terms of internet of things. This is one of the reason why mqtt is used most in internet of things.
So talking about IoT, a topic can be a sensor value or a parameter which is being sent by one or more devices and which needs to be received by one or more devices. So topic names can be
- Temperature
- Humidity
- Pressure
- Presence sensor
- LPG Sensor Value
- and so on…
topic can also be the device control action. For example, in IoT you have to turn on cooler or heater or a fan or a motor or anything using a relay. Then topic names can be
- Relay 1 or relay 2 or so on
- LED
- Fan
- Motor
basically topic is a means of communication with which the clients will communicate each other.
Now the clients or individual computer in mqtt, can have one of the 2 jobs.
- Sending Data
- Receiving Data
- Doing both
so depending on whether its sending data or receiving data, the client computers are called as
- Publisher (who sends data)
- Subscriber (who receives data)
Look at the generalized mqtt block diagram
As you can see, the entities are now called as publisher and subscriber. As the storage is completely optional at the server, its now called as broker. The broker is nothing different than server in our previous case. It is a physical entity in your network, like a computer PC or raspberry pi. But saving data ont this computer is optional. The communication between devices happens over a particular topic. Lets assume there is just temperature data which needs to be exchanged. In this case, lets see how mqtt communication happens between 2 computers.
So the subscriber here, at the time of boot up or at the time of running the application code, has to subscribe to the topic it wants to read from the network. Lets assume the topic name here is temp. The publisher is supposed to publish to this topic to broker. And the subscriber has to stay subscribed to this topic from broker. The communication happens like this.
- Publisher will read the temperature value and publish over the topic called temp
- Broker receives the temperature value over a topic called temp and immediately distributes / sends the value to all the subscribers of the topic temp
- All the subscribers of topic temp receives the value.
Its just pure simple and fun to make communication this way. The best part is, the publisher can publish to several topics and subscribe to several topics as well. The same case with subscriber.
How to use mqtt in python
Having understood the theory of how mqtt works, lets give it a quick spin. For this experiment, you’ll need at least 2 computers to have real fun and experience of understanding communication. One of our computer will act as broker and subscriber. One computer will act as publisher.
Installation and Running MQTT python program is a 2 part process.
- We need a MQTT server who runs the service, technically, its called mqtt broker — Its Raspberry pi in this experiment.
- We need a publisher, who just sends some data to or receives some data from the server, — Its PC in our case can be other RPi or Nodemcu or esp32 or any such thing. In current scene, its the PC
So what we need is an MQTT server (or broker) running on raspberry pi. mosquitto is one such server which can run on Raspberry pi. In this tutorial, we use mosquitto server on raspberry pi.
Now lets see installations to be done
Install Python MQTT Library On raspberry Pi
we need to install both the mqtt server as well as the python client library on raspberry pi so that we can not only receive data, but also be able to interpret it using python code. In short to receive the data in python codes.
so on raspberry pi, first install the mqtt server with below command. Make sure you type these commands in terminal of your raspberry pi computer.
sudo apt-get install -y mosquitto mosquitto-clients
this command will install the mosquitto MQTT broker and client libraries on raspberry pi and will run it automatically in the background. The broker is now installed on the raspberry pi. Now lets run the mqtt broker so that its operational. This can be done by
sudo systemctl enable mosquitto
Above line will run the broker on raspberry pi.
Next, we need to run the subscriber code on raspberry pi as well. This code will read the values published by the publisher. In this case, we’re making use of broker itself as a subscriber as well. To access the mqtt functionalities through python program, we need to install the mqtt python libraries on raspberry pi. Mosquitto here is the software which will act as broker meaning it will act as the receiver of data or sender too in some cases. To read the received data in python program, we need python mqtt library. To install python MQTT library, run below command on termianl of raspberry pi. The most popular mqtt library for python is paho-mqtt library which can be installed with this command
sudo pip install paho-mqtt
Above line will install the paho-mqtt python library with which we can write a code to read the data coming from a sender like PC or microcontroller.
If you’ve both python2 and python3 installed on your raspberry pi, which you obviously would have; then its better to explicitly specify that you want to install libraries for python3 with this command
sudo pip3 install paho-mqtt
Above command will install paho mqtt libraries for python3
MQTT Python Code
Now the Code, below python code is useful to receive data on raspberry pi mqtt broker on specified port address. As discussed before, MQTT needs a topic like temp to receive data, this is also called as channel, so whatever topic you’re specifying here, should be the same one used on the sending device (publisher). Look at the code below, this is a python code for raspeberry pi to receive mqtt data.
import paho.mqtt.client as mqtt #import library
MQTT_SERVER = "localhost" #specify the broker address, it can be IP of raspberry pi or simply localhost
MQTT_PATH = "test_channel" #this is the name of topic, like temp
# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc):
print("Connected with result code "+str(rc))
# Subscribing in on_connect() means that if we lose the connection and
# reconnect then subscriptions will be renewed.
client.subscribe(MQTT_PATH)
# The callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):
print(msg.topic+" "+str(msg.payload))
# more callbacks, etc
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
client.connect(MQTT_SERVER)
client.loop_forever()# use this line if you don't want to write any further code. It blocks the code forever to check for data
#client.loop_start() #use this line if you want to write any more code here
Install MQTT python libraries On PC
After installing everything required on Raspberry Pi, we’ve to install the things needed on PC
We only need the python mqtt libraries on PC for sending data to mqtt broker running on raspberry pi. It can be installed by running below command in command prompt of windows. Make sure you run the command prompt by right clicking and clicking on “Run as Administrator”
pip3 install paho-mqtt
make sure, you open the command prompt as “Administrator” otherwise it may cause issues in installation.
Now use below simple python code to send data to the test channel created in raspberry pi mqtt server. You need to specify the raspberry pi IP address as the mqtt server address in the below code. Run below code on PC
import paho.mqtt.publish as publish
MQTT_SERVER = "192.168.1.5"
MQTT_PATH = "test_channel"
import time
while True:
publish.single(MQTT_PATH, "Hello World!", hostname=MQTT_SERVER) #send data continuously every 3 seconds
time.sleep(3)
And that is about it. whatever data instead of “hello world” you send, it’ll be sent and published to the mqtt broker. Now this is a very simple and crude sample and as you learn more, you can experiment more about it.
Interesting thing is, MQTT is a preferred way to send data from any kind of IoT Device to major cloud service providers like microsoft azure and aws.
Keep Experimenting!!!
Excellent article sir,
I just wanted to know is there any requirement for rpi to connect to the internet for the above exercise?
Hi Sampath, no if u just want to test mqtt between pi and pc, u don’t need internet, however you’ll need it while installing packages
Sir I want to display dashboard in internet using mqtt protocol.Can I done the project using rpi 4 alone?
Yes, it can definitely be done
Hi Amit,
Your tutorial was very helpful for me to implement and understand mosquitto. I have setup everything according to your tutorial. I am able to publish data from my pc to the RPi. But, I am facing problems in sending the data from the RPi to the PC. Please guide me in how to send data from RPi to PC.
Thanks in advance.
BR,
Kevin
I know it’s too late
Can you write me on Amit at kitflix dot com?
Hi Kevin
whats the issue you’re facing?
Hai, the way of explanation is awesome. Can you please explain how to send the sensors data value in live visualization to the subscriber?
Sensors can send live data to mqtt broker, which in turn updates all the subscriber. In a way, the sensor node (publisher) has no direct contact/access to the subscriber (the one which receives the data )
hope that helps 🙂
You can send feeds to subscriber, to show live visualisation, you’ll have to create a dashboard with a service like adafruit IO
Hi Amit!
many thanks for the very well explained material. I am facing some issues on establishing the server on my raspberry pi. I keep getting a “ValueError: Invalid Host”, both with localhost and the actual IP, the same happens also on a local non-internet connected wifi.
Many thanks in advance!
Its quite late to reply, but just let me know if its solved?
There were too many spam comments and hence your comment was left from responding, sorry :/
Can U share a screenshot?
Are You positive that mosquitto is installed??
A screenshot of the code would be much better
its difficult reading in comment section
I’m having some trouble with running the MQTT Broker on the pi.
There seems to be an error when using the loop_forever() function.
(Hopefully this screenshot link will work: https://imgur.com/a/da7viL9)
The error is “Invalid host.” when reconnecting. The IP address for the pi is definitely correct as it is what I am using to control the pi over Putty.
Have you got any idea where I might be going wrong?
Thanks a lot!
I hadn’t set up the broker correctly.
I launched it using the details here and it worked (https://appcodelabs.com/introduction-to-iot-build-an-mqtt-server-using-raspberry-pi)
Key thing is to have the Broker running in a different terminal
Great Thing 🙂
I run the link you shared. it was working.
What did you change here that you did not receive the invalid host error anymore?
Its some issue with broker
Your code is very understandable but can you please suggest me if I want to send message signal between two raspberry pi’s connected to two different network , how could I do that..?
there are multiple way’s to go with it. I think the simplest will be to send data using IP address and socket based coding. You don’t need any servers to be running for that. Check this https://stackoverflow.com/questions/40228915/sending-data-back-and-forth-from-two-raspberry-pis-with-socket-library
If at all you need to use mqtt for this, then setup mosquitto on one raspberry pi, and send the topics from other raspberry pi. And receive subscription updates from server.
So the server has
1. Mosquitto broker
2. 4 Topics, 2 which other raspberry pi is sending
3. 2 topics which this broker will update
4. Python script on other raspberry pi to send first data to first 2 topics and receive data of last 2 topics
5. Python script on the raspberry pi on which broker is installed to update the last 2 topics and receive first 2 topics
If it doesn’t make sense (which I don’t think it is 🙂 ) then ping me on amit at kitflix dot com, I’ll clarify
Thank you for reading 🙂
Hello, thanks for this wonderful tutorial.
I have set up the broker in my laptop (Windows 10).
“Connections only be possible from clients running on this machine”
“Create a configuration file which defines a listener to allow remote access.”
I am not able to access the Broker from other device. (Example: IOTMQT-Panel App,/ MQTT dashboard app)
What should I change in the configuration file to allow remote access.
Mosquitto version 2 has made these changes that you can’t access or listen to remote devices. I’ll be updating this tutorial soon.
meanwhile, please check this one
http://www.steves-internet-guide.com/mosquitto-broker/
much more detailed than what I’ve written 🙂
Thank you very much, for the quick reply.
Your work is helpful to many, keep continuing.
Thank you 🙂
Faced an issue with “Invalid Host” and started debugging, and found that client.connect(MQTT_SERVER) is missing in the code @ line 21.
Tested it and it works fine for me 🙂
Thanks for the detailed blog!!
Thank you for suggested edit, its updated 🙂
Hello Amit, thank you for this tutorial!
I’m having troubles on the publisher side, I’m having as result: ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it.
How do I solve it?
Hi Raphael, can you paste your exact code and send a screenshot of error?
have you checked that the mosquitto broker is installed?
Bonsoir Mr Amit Rana,
S’il vous plait, quel moyen me conseillez vous pour un échangé de données (flux vidéos + données capteur) bidirectionnel entre une application sur PC (Windows) et un Raspberry Pi 4B.
Merci d’avance.
Hi,
I’m no expert in spanish 🙁
But what I can translate, I can see you’re asking about bidirectional data transfer between pc and raspberry pi. And you would also love to share videos.
In this case, socket communication will do best. There are some trials of sending video feed from raspberry pi to pc using flask if you search on internet. The socket connection will be best because speed limitations won’t be there
Let me know if it helps 🙂
Thank you
So can we make communication between MQTT Mosquitto broker and the phpMyAdmin database
Yes, its possible, but can’t say direct communication from mqtt broker to phpmyadmin db, you’ll have to write a bridging python script which receives the information from Broker and writes to database. This way, yes its possible
hello Amit i am reading RFID reader data with raspberry pi and i need to setup MQTT broker server so i can receive data from Raspberry pi can you help me out?
Let’s say I created a access point with rooter and then I can connect both raspberry and another system(like ground station computer raspberry pi ) to the rooter. Can the system communicate with each other ? Like I want to send sensor data from raspberry to server and then I want to see the data from another system but I don’t want to use internet . I am a student and I am trying to make some projects. So , if you can help me I will be preciated for your help.
Hi, yes what you’re saying is totally possible.
You can connect raspberry pi to a router and you’ll need to setup some kind of communication server on this device. For e.g. if you’ve setup mqtt server on this raspberry pi, then you can receive messages from other raspberry pi on the first one without connecting to internet.