We need a structured list of all messages and commands

GertSt · April 14, 2020, 9:56am

I think Velbus has the best view on how we best set up the structure, as you guys must already have something simular for VelbusLink, right?

So may be Velbus can give us a structure to start from?

I’m also willing to help with this.

Stef_Coene · April 14, 2020, 3:43pm

@VEL524 I can document the data structure I need in my scripts.
This will give you an idea for at least one project what kind of data is needed to interface with Velbus.

I can extract with almost 100% accuracy the needed data from the protocol files.
I can generate a json file with this data. Maybe this can kickstart or help in speeding up the release of the protocol data in json format?

Stef

Stef_Coene · April 14, 2020, 9:14pm

@VEL524: I already documented the data layout I use in my scripts.
See this document, starting from ‘4. Files’:
https://docs.google.com/document/d/e/2PACX-1vQtVpozXUgypN6_IAmaaVj0cxtqXUYm1gj_n4nqt8vWeznqdSgb8g1LCHe_bkZmlg0-WkLHiYUKFa7S/pub#h.sv4xgu4dn1mg

It looks complicated and it is, but that’s just the result of all the possible module types and channels.
I tried to reuse as much code as possible.
If I can be of any help, let me know!

Stef

GertSt · April 15, 2020, 7:20am

Wow Stef, that’s very useful information!

Going through this doc, the real value to have an overview is obvious: We are able to find out the overlapping, generic parts that apply to several modules and thus can be reused.
You already did a great job on this!

Thanks for sharing!

cereal · April 15, 2020, 8:54am

is all of this coming from the pdf files?
or is some data added manually?

Stef_Coene · April 15, 2020, 12:05pm

Both.
Velbus_data_protocol_auto.pm is the output of pdf2txt.pl and is 100% automatic (except for some types in the protocol files) generated.
The other Velbus_data_protocol_*pm files contains manually added information mostly based on the protocol files that I needed for my scripts. The file names speaks for them self: channel, memory and message information.

Some stuff is missing because I’m not interested in in it.

Stef

GertSt · April 15, 2020, 1:41pm

Would it be possible to already create a JSON file with the same data? Of course only if it’s not a lot of work! That would already be a very good start that would benifit a lot of us! (To be honest, at least me! )

I don’t see the databytes info in your file. Is this because you don’t need the data for your project or because it’s not easy to extract?

GertSt · April 16, 2020, 6:08pm

@VEL524:
This list from Velbus shows a well structured overview of all commands, but does not contain protocol info. May be this can be used as a starting as well?

Gert

VEL524 · April 17, 2020, 7:48am

Perhaps that can help if it isn’t too much work for you to generate that.

I like how you are dealing with commands. Instead of implementing them exactly as they are mentioned in the protocol manuals, you just watch the bytes you need. Very clever.
If this can be turned into a data file, we might be able to prevent that integrations need to implement every message of every module in their coding language. I’m going to call them “features” as now, perhaps somebody may have a better concept name.

Could you explain a bit how you are dealing with more complex messages? I see you are using “Convert” there and I assume that you are putting that logic into code. It would be cool if that could take place in the datafile as well somehow.

Stef_Coene · April 17, 2020, 2:44pm

I already forked the github repository: https://github.com/StefCoene/moduleprotocol
I will add my code to parse the pdf files.

@VEL524: I will also document the code in more detail. Including the ‘Convert’ option.

Stef

Stef_Coene · April 18, 2020, 6:21pm

Hi,

I updated my github repository: https://github.com/StefCoene/moduleprotocol

I added my parsing script and the other data files I have. The result is a json file:

github.com

StefCoene/moduleprotocol/blob/master/out/protocol.json

{
   "ChannelTypes" : {
      "Blind" : {
         "Get" : {
            "Message" : "EC"
         },
         "Modules" : "03,09,1D,2E",
         "Set" : {
            "Match" : {
               "DOWN" : {
                  "Message" : "06"
               },
               "STOP" : {
                  "Message" : "04"
               },
               "UP" : {
                  "Message" : "05"
               },
               "\\d+" : {
                  "Action" : "POSITION",

This file has been truncated. show original

I also added a README with extra information.

Warning: I changed the script that I use to parse the protocol files. So I hope I didn’t made a mistake.

Stef

GertSt · April 18, 2020, 7:26pm

Wow Stef, what you did here is unbelievable!

It took me some time to fully understand the structure, but after going through it, I’m very impressed by the details and the well-structured sections.
It’s clear that you already spend a lot of time structuring those protocol files!

Thank you very much for your time and for sharing this. I think other developers can gain quite some time having this info available.

How do you see going further with this? May be @VEL524 can shed his light on it?

Gert

MDAR · April 18, 2020, 7:36pm

Or her…

Who knows…

Stef_Coene · April 18, 2020, 7:40pm

I already found a few bugs. I once asked if I can get a module of each type so I can debug them all, but they never gave me them

And only the stuff I needed for my scripts are included in the json. And in the format that suited me the best.
Let me know if you mis anything and I will try to include it.

Stef

GertSt · April 18, 2020, 7:41pm

Velbus, I think the time is right to reveal the real persons behind those numbers!

GertSt · April 18, 2020, 8:40pm

I think you deserve it Having the OpenHab interface adds quite some value to Velbus.

It’s a good start. A lot of commands can still be added. We just have to think on how the community can help with that. Because once values are added manually we can not regenerate it anymore.

Adding them manually is not that easy either because of the linked sections.
We might create a dedicated editor for it, including data validations and revisions… Hmm, that might work…

The purpose of having a JSON like this is to simplify sending/receiving bus messages in whatever language, by having that higher-level, structured metadata.

It should be simple to

Use this metadata to filter out modules and messages to easily generate the right Velbus packet data. This can then be translated to whatever underlaying programming language.
Add new modules and messages to the JSON file when Velbus releases new modules. So versioning is important.

With that in mind, I should make a ‘simple’ test app that integrates the JSON to see how this structure works out. But as I said, it looks promising.

More commands and detailed data byte info are of course welcome, but that’s a hell of a job. Not sure if more can be parsed out of the PDFs? That’s why we need to think about a community driven way to do this.

Looking forward to both the opinions of @Stef_Coene and @VEL524 about this.

MDAR · April 18, 2020, 8:55pm

I can’t give any hardware, but you’re both welcome to remotely connect to my networks to test your solutions.

I’m only a couple of modules short of the full ( current ) range.

Stef_Coene · April 18, 2020, 9:35pm

I just wrote some scripts for fun
And published them on github because an ex-collegae was interested in them.

The json file is just the end result.
The information comes from the protocol files, but mostly from the files in the data directory. These *.pm files are in perl format. But they contain the most important data and some logic to reuse some code.
Don’t be scared about the perl format, it’s actually not really perl code, but a mult-hash definition. I can convert them to json, but I think the perl code is easier to read then json code.

I recommend to not change the json file itself, but the *.pm files and regenerate the json file.

With format I didn’t mean json. But more the way the data is stored in the json file.

I tried to explain how I use the data in the json in README-StefCoene.md, but that’s not finished yet.
For instance, the ‘Get’ and ‘Set’ are hard to explain, even for me
I need to reread my code to find out how the data in the json file is used.

It’s easier to just read the pdf’s, update the *.pm files and regenerate the json file.
But I think most of the interested stuff is already in my code. I’m pretty sure I can parse all channel update messages and control all channels with my scripts.

I skipped everything related to alarms and programs because I don’t use them in my setup. I don’t know if there is any need for it?

Stef

cereal · April 19, 2020, 5:29am

This is indeed a great start, or even an awsome start.

the only thing what i miss is that in some definitions there are no modules listed, is this a bug?
about the basedon, i think if we want to make this generally avilable (an only generate the json) it would be better to just duplicate the data.

And would it be possible to add the modulename to the ‘ModuleTypes’ object?

the buttoncounter type is missing some info

MDAR · April 19, 2020, 7:50am

This is an odd one.

There seems to be a clear 50 / 50 divide.

Some people don’t seem to want any kind of timed events within Velbus and use some other kind of scheduler or just manual adjustment.

Then the other people (once they discover the Velbus alarm times) use them for full control of their property.

Changing program groups is again something that isn’t asked for until people get very comfortable with the advanced configuration.

The Alarm Clock widget I’ve adapted for openHAB2 HabPanel certainly makes it easier to interact with them.