Node Red & MQTT for Annex?

Place your projects here
Post Reply
peridot
Posts: 56
Joined: Mon Mar 08, 2021 4:54 am
Has thanked: 9 times
Been thanked: 106 times

Node Red & MQTT for Annex?

Post by peridot »

I have a mixture of ESP devices fulfilling various HA tasks, using Annex, ESPEasy, Tasmota, then these feed into Node Red and a MQTT broker running on a Raspberry Pi V4. The I use https://play.google.com/store/apps/deta ... IOT Panel running on Android for Display and Interaction.
I have been tinkering for while wondering if i could replace the RPI with just an Annex device but not wanting to use a cloud MQTT Broker I was a a bit stuck.There is however a tiny MQTT Broker that will run on a ESP8266 and I have played a couple of times and found it very good but it was limited in TCP connections so I didn't go further. https://github.com/martin-ger/esp_mqtt

Since the release of Annex v 1.60 I thought I would have another go and wow I like v 1.60, great stable UI , so I worked further on some code I had developed and thought If i coupled an ESP32 running Annex with an ESP8266 running the MQTT broker and the Annex Mini IOT was born.

The Annex software I have called "WireTool" as I try to replicate the Node Red software the allows you to Wire devices together.
The Basic concept is I use UDP transmissions from my various sensors sending Json strings. Annex receives the UDP and posts MQTT Topics to the Broker. Annex also subscribes to a single Topic called "wtout" (json stinng) on the broker for outgoing commands, Annex can then parse the json string and send commands out to the ESP devices via UDP or HTTP as required.
This approach only needs two TCP connections to the MQTT for a fully functioning system. This is well within the viable 8-10 TCP connections available on the broker leaving plenty of connections for other connected MQTT displays etc.

The Annex "Server" also created a "feed" for each sensor value in each topic and saves them periodically (say 30secs) to SD as a time based data series CSV file which may then be accessed late for ploting etc. A new file is created daily and filed in folders folders for Day/Month/Year.
A "Task" based timer is also provided for On / Off trigger functions.
The MQTT Broker may be programed and its status monitored via a serial connection.

I hope this may have been of interest to some of you , if so I will post the code and update as developed.
IMG_20240917_182713091.jpg
IMG_20240917_182151240.jpg
You do not have the required permissions to view the files attached to this post.
User avatar
cicciocb
Site Admin
Posts: 2783
Joined: Mon Feb 03, 2020 1:15 pm
Location: Toulouse
Has thanked: 592 times
Been thanked: 1990 times
Contact:

Re: Node Red & MQTT for Annex?

Post by cicciocb »

Yes, it seems a very interesting project so, please, share it as I'm sure that many will be interested about it :idea:
TRS80_MKI
Posts: 21
Joined: Sun Apr 18, 2021 3:38 pm
Has thanked: 11 times
Been thanked: 5 times

Re: Node Red & MQTT for Annex?

Post by TRS80_MKI »

Hi ,
very interesting project .
I use something like your project to control a turning table and 4 cameras to make photogrammetry of object up to 100 kg.
i use a esp32 s3 , hostin a web server with a ui to control all process , this webserver is a captive portal done with framework arduino on platformio.
Along the server a MQTT Broker running very smoothly (i use node-red to debug it...) :https://github.com/mlesniew/PicoMQTT
the setup is :
-- esp32 s3 Uno board :: n16r8 - webserver
-- mini S2 to control 4 cameras (use MQTT client)
-- mini S2 to control stepper motor(serial servo) according number of pictures to be processed (use MQTT client).

I'm translating control process (s2) in Annex32

I'll post some pict. when project go live ...

@+
TRS80_MKI
peridot
Posts: 56
Joined: Mon Mar 08, 2021 4:54 am
Has thanked: 9 times
Been thanked: 106 times

Re: Node Red & MQTT for Annex?

Post by peridot »

Update.
Here is current working code

Core Functions
UDP, HTTP, ESPNow network sensor feeds. Upto 30.
All feeds logged to day file (CSV) stored in Month/Year folder stricture.
processed and published to local MQTT Broker.
Comprehensive Task timer, triggers events, period, sequences, On/Off , recurring etc.

User Functions (Like..)
Home Energy
Weather data

Eg. Current code includes routine to parse weather sensor data and create file for submitting to https://cumuluswiki.org/a/Software#What ... s Weather . I have Cumulus running locally on a Synology NAS.

Images for CumulusMX Dashboard and data log
2024-10-13 13_07_36-Dashboard - Cumulus MX — Mozilla Firefox.png
2024-10-13 13_09_44-Mozilla Firefox.png

Code: [Local Link Removed for Guests]

'
' ' *******************************
'    Date: Aug 2023
'    Filename: wiretool2-xx
'    Function:
'    Device: ESP32
'
'    References:
'
'  *******************************
'v0.19 rework version, add tasks0.7
'v0.20 rework version, add tasks
'v0.21 update from previous
'********************************************************************************************
'Constants & Variables
ver$="v0.21"



LastReset=BAS.RESETREASON
myIP$ = WORD$(IP$,1)   'return local IP address Module
'Declare APs and Passwords

'
'IO hardware
'wifi_enable=pin12

'LEDs
redled=17     'power led
grnled=21   'network led
yelled=15   'activity led
butt=32   'button
PIN.MODE grnled, OUTPUT
PIN.MODE yelled, OUTPUT
PIN.MODE redled, OUTPUT

'control
'initialise IO
'set leds off
pin(grnled)=1
pin(yelled)=1
'set power led on
pin(redled)=1

pause 100 'settle time
'Global: Variables and Array
tfeeds=30 '# of feeds
dim feeds$(tfeeds)
dim fvalue$(tfeeds)
nfeeds=tfeeds
header$=""
'task Var
task_res=1000  'task resolution _default 1000ms
task_max=30 'max tasks
task_flag=0 'set any time a task goes to zero to run task
load_flag=1 'set any time to allow tasks to load without tasks running
dim tt(task_max)  'Task Timer array
tx=0:ty=0

'setup initial tasks
task 0,0,1,"","" ' usually run once on startup
timer0 task_res ,T1   'establish seconds "Task Timer"
'system variables
tx=0:y=0:m=0:ret=0:ix=0
netok=0
wifi_enabled=1
it=0: rt=0 'cpu timings
day$="":month$="":year$="":lastday$=""
c$=",":q$=chr$(34)
js$="":utx=0
cr$=chr$(13):lf$=chr$(10)
outdoor$="0"
indoor$="0"
'Setup and Initialise
' Timers & Interupts
'Network etc
wlog "init espnow " ; espnow.begin  ' should print 0 if all OK
onEspNowMsg message ' set the place where jump in case of message reception
udp.begin 5001  'set the UDP commmunication using port 5001
ONERROR GOTO Error_Handler
onudp gudp
onUrlMessage urlAjax
ONWGETASYNC answer_done
serial.mode 115200
option.WDT 10000    ' set the WDT at 10 second
gosub initfeeds
wlog mqtt.setup("mqtt://192.168.0.85:1883", 1) ' set debug
wlog mqtt.connect("", "")
'wlog mqtt.subscribe("wtout/#")
'onmqtt mqtt_event
pause 1000
wlog "Annex-wiretool "+ ver$;" ";ramfree

main:
do
  it=millis
  option.WDTreset
  'Tasks are run when expired tasks are ready
  
  'if 24Hr time is loaded then the Task time will be calculated
  'for today or next day depending on time.
  if task_flag=1 or load_flag=1 then
    ' wlog "task loop"
    task 0,0,-1,"",""
    task 1,11,0 ,"10:00","15:45" 'start
    task 11,0,0 ,"15:45","" 'stop
    task 22,0,0,"12:15","" 'active
    task 23,0,10,"","" 'active force repeat task every 60 secs
    ' task 24,0,30,"","" 'Store
    task 30,0,0,"00:00",""' trigger @ midnight
    task 3,0,10,"",""'
    
    task_flag=0 'reset flag
    load_flag=0 'reset flag
  endif
  
  '********************************************
  ' insert your main code here....
  '********************************************
  
  rt=millis
  ' wlog rt;" ";it
  rt=(rt-it)
  
loop
'***** End main program loop
end


'Core_Subs:

sub splitDate
  day$=str$(Val(left$(date$,2)))
  month$=str$(val(mid$(date$,4,2)))
  year$="20"+right$(date$,2)
end sub

'pulse PwrLed
sub pledred
  pin(redled)=0
  pause 25
  pin(redled)=1
end sub

'pulse Netled
sub pledyel
  pin(yelled)=0
  pause 25
  pin(yelled)=1
end sub

'pulse Actled
sub pledgrn
  pin(grnled)=0
  pause 25
  pin(grnled)=1
end sub

'read in feeds data
initfeeds:
for ix= 1 to nfeeds+10
  Ret$ = trim$(FILE.READ$("/cfg/feeds.csv", ix))
  if left$(ret$,1)<>chr$(35) then
    feedname$=word$(ret$,2,c$)
    pos=val(word$(ret$,1,c$))
    'wlog pos;feedname$
    feeds$(pos)=feedname$
    header$=header$+feedname$+c$
  else
    nfeeds=ix-1 'comment line
  end if
next
return



sub getjson(j$,r$,w$)
  local x$,dc
  x$=json$(j$,w$)
  if x$ = "not found" then exit sub
  r$=x$
end sub

sub getfeeds(j$)
  local f$
  for gx=1 to nfeeds
    f$=json$(j$,feeds$(gx))
    if f$ <> "not found" then
      fvalue$(gx)=f$
      'wlog j$
      'wlog feeds$(gx)
      'wlog f$
    end if
  next
end sub


'inputs data from UDP
gudp:
v$ = udp.read$
'pause 100
wlog "UDP:"+v$
Topic$=word$(v$,1,"=")
payload$=word$(v$,2,"=")
ret=mqtt.publish(topic$,payload$)
getfeeds v$   'load feed data
pledgrn
store v$
v$=""
return

'inputs data from HTTP
ghttp:
h$=UrlMsgGet$("fulljson")
'wlog "http";h$
pledgrn

h$=""
return

'inputs data from ESPnow
espnow:
'h$=UrlMsgGet$("fulljson")
''wlog "http";h$
'   pledgrn2

' h$=""
return

sub store(sv$)
  local sx,s$
  splitdate
  'wlog day$;month$,year$
  for sx=1 to nfeeds
    s$=s$+fvalue$(sx)+c$
  next
'make a file header on new file
 ret=FILE.EXISTS("/data/"+year$+"/"+month$+"/day"+day$+".csv")
 if ret=0 then
  msglog "/data/"+year$+"/"+month$+"/day"+day$+".csv",header$
 end if
  'pick out json json$,result$,name$
  'wlog "/data/"+year$+"/"+month$+"/day"+day$+".csv",s$
  msglog "/data/"+year$+"/"+month$+"/day"+day$+".csv",s$
  getjson sv$,outdoor$,"tempc"
  getjson sv$,indoor$,"inTemp2"
  pledyel
end sub

urlAjax:
''wlog "message received " + UrlMsgGet$("a") + " " + UrlMsgGet$("b")
return

'task run tsk=Task#, atsk=associated task,lsk=Task time, Ts24$=24hrTime/"",Tf24$=24hrTime/""
sub task(tsk,atsk,lsk,ts24$,tf24$)
  local ret$,st,ft
  'wlog "Task";tsk;" ";tt(tsk);" ";lsk
  if tsk>task_max then exit sub 'if > than Max tasks available
  if tt(tsk)<0 then exit sub  'if task time has ended
  
  'check for associated task (finish time)
  'check if task has start and finish
  'and if start time has passed allow task to run
  if (atsk>0) and (ts24$<>"") and (tf24$<>"")   then
    ''has start time passed
    if (timeunix(time$)>= timeunix(ts24$)) and (timeunix(time$)<=timeunix(tf24$)) then
      tt(tsk)=0
      wlog "Just Missed start time, so run start task"
    else
      tsk=atsk
      tt(atsk)=0
      wlog "Missed start & finish time , so run finish task"
    end if
  end if
  
  if tt(tsk)>0 then exit sub  'if task time is still running
  if load_flag=1 or task_flag=1  then 'OK to Run Tasks
    'wlog "Test Tasks"
    select case tsk 'task number
      case 0 : wlog "Run OnStartup"
        gosub onStartup
        tt(0)=-1
      case 1 : wlog "On "':WGETASYNC("http://192.168.0.42/tools?cmd=GPIO,12,1")
      case 2
      case 3 : gosub  cumulus
      case 4
      case 9
      case 5
      case 6
      case 7
      case 8
      case 9
      case 10
      case 11:wlog "Off "' WGETASYNC("http://192.168.0.42/tools?cmd=GPIO,12,0")
      case 12
      case 13
      case 14
      case 15
      case 16
      case 17
      case 18
      case 19
      case 20
        
      case 21
      case 22 :wlog "11:45 Task"
      case 23
      case 24: 'store 'store feeds @ 30 secs
      case 25
      case 26
      case 27
      case 28
      case 29
      case 30
        '
      case else
        tt(tsk)=-1 'stop the task
        
        
    end select
  end if
  ' now set up new task time
  if lsk>0 then
    tt(tsk)=lsk 'reload task time
  else
    if ts24$<>"" then
      tt(tsk)=timeunix(ts24$)
      if tt(tsk)<= timeunix(time$) then
        tt(tsk)=tt(tsk)+(86400)
      else
        tt(tsk)=tt(tsk)-timeunix(time$)
      end if
    end if
  end if
  wlog "new task time to run in secs for Task "+str$(tsk)+" is: "+str$(tt(tsk))
end sub

T1:'Task Timer
for tx= 0 to task_max
  ty=tt(tx)
  if ty>0 then
    ty=ty-1
    if ty=0 then
      task_flag=1
      'wlog "set ";tx
    endif
    tt(tx)=ty
  end if
next tx
return



sub msglog(f$,msg$)
  file.append f$,date$+","+time$+","+msg$+chr$(13)+chr$(10)
end sub

sub createfolders()
  for y= 2022 to 2028
    for m=1 to 12
      'file.append "/data/"+str$(y)+"/"+str$(m)+"/data.csv", date$+time$+","+"12345.10"
      'ret=file.delete("/data/"+str$(y)+"/"+str$(m)+"data.csv")
      Ret = FILE.MKDIR("/data/"+str$(y)+"/"+str$(m))
    next m
  next y
end sub

'
onStartup:
wlog "Running OnStartup"
select case BAS.RESETREASON
    wlog "case ";BAS.RESETREASON
  case 0: msglog "/logs/log.txt","Unknown"
  case 1: msglog "/logs/log.txt", "Power on reset"
  case 2: msglog "/logs/log.txt", "Unknown"
  case 3: msglog "/logs/log.txt", "Software reset digital core"
  case 4: msglog "/logs/log.txt", "Legacy watch dog reset digital c"
  case 5: msglog "/logs/log.txt", "Deep Sleep reset digital core"
  case 6: msglog "/logs/log.txt", "Reset by SLC module, reset digital core"
  case 7: msglog "/logs/log.txt", "Timer Group0 Watch dog reset digital core"
  case 8: msglog "/logs/log.txt", "Timer Group1 Watch dog reset digital core"
  case 9: msglog "/logs/log.txt", "RTC Watch dog reset digital core"
  case 10: msglog "/logs/log.txt", "Intrusion tested to reset CPU"
  case 11: msglog "/logs/log.txt", "Time Group reset CPU"
  case 12: msglog "/logs/log.txt", "Software reset CPU"
  case 13: msglog "/logs/log.txt", "RTC Watch dog reset CPU"
  case 14: msglog "/logs/log.txt", "for APP CPU, reset by PRO CPU"
  case 15: msglog "/logs/log.txt", "Reset when the vdd voltage is not stable"
  case 16: msglog "/logs/log.txt", "RTC Watch dog reset digital core and rtc module"
  case else
    msglog "/logs/log.txt", "Code-"+str$(BAS.RESETREASON)
end select
return



sub status
  local s$
  s$=s$+bas.ssid$+c$
  s$=s$+str$(wifi.rssi)
  pause 500
  msglog "/logs/status.txt",s$
end sub

Error_Handler:
wlog "Error text:"+BAS.ErrMsg$+" ,Error num="+str$(BAS.ErrNum)+" ,Error line="+str$(BAS.ErrLine)
msglog "/logs/err.txt","Error text:"+BAS.ErrMsg$+" ,Error num="+str$(BAS.ErrNum)+" ,Error line="+str$(BAS.ErrLine)
Return ' returns to the line following the error

mqtt_event:
'wtout$=mid$(mqtt.topic$, 7)
wlog "TOPIC  : "; mqtt.topic$
wlog "MESSAGE: "; wtout$
pledyel
return

answer_done:
wlog WGETRESULT$
Return

sub stampUnix()
  stampunix=dateunix(date$)+timeunix(time$)
end sub

message:
wlog "RX:"; espnow.read$; " from "; espnow.remote$ ' print the message
return

'usage  getjson newline$,pw$,"iwatts"
sub getjson(j$,r$,w$)
  'wlog j$;" ";r$;" ";w$
  local x$,dc
  x$=json$(j$,w$)
  if x$ = "not found" then exit sub
  r$=x$
end sub


cumulus:
wlog "Run Cumulus"
utx$=str$((dateunix(date$(1))+timeunix(time$))-3598,"%u",1)
'wlog utx$
js$="{"+q$+"units"+q$+":{"+q$+"temperature"+q$+":"+q$+"C"+q$+c$+q$+"windspeed"+q$
js$=js$+":"+q$+"kph"+q$+c$+q$+"rainfall"+q$+":"+q$+"mm"+q$+c$+q$+"pressure"+q$+":"+q$+"hPa"+q$+"}"+c$
'unix time
js$=js$+q$+"lastupdated"+q$+":"+utx$+c$
'temperature
js$=js$+q$+"temperature"+q$+":{"+q$+"outdoor"+q$+":"+outdoor$+c$+q$+"indoor"+q$+":"+indoor$+c$
js$=js$+q$+"dewpoint"+q$+":"+str$(99.9)+"}"


js$=js$+"}"


WGETASYNC("http://192.168.0.17/api/updatefile-01.php?key=aa678123w&file=weather&json="+js$,80)
Return



'User_Subs:




'HTML_UI:
You do not have the required permissions to view the files attached to this post.
lyizb
Posts: 241
Joined: Fri Feb 12, 2021 8:23 pm
Has thanked: 121 times
Been thanked: 84 times

Re: Node Red & MQTT for Annex?

Post by lyizb »

Great project. What devices to you have variously using MQTT, UDP, and ESP-Now? That is, what does your home network look like?
peridot
Posts: 56
Joined: Mon Mar 08, 2021 4:54 am
Has thanked: 9 times
Been thanked: 106 times

Re: Node Red & MQTT for Annex?

Post by peridot »

I have a various sensors involved in weather data collection , a garden PV power supply with sensor monitoring, a greenhouse with hydroponic monitoring sensors. Those are the main areas I guess but also I control a water fountain pump and other tasmota controlled smart switches etc.

Previously I have used Raspberry Pi running Node Red and Mosquito and dabbled with Home Assistant but had kept coming back to see if I could come up with a much simpler hardware design. I also run Cumulus Weather on a Synology NAS.

Annex has now become such a comprehensive and stable package (Thanks to Francesco !) that a simple but "Super" Home Automation system should be possible.

Mike
Post Reply