CoRoutines library for Arduino

[Isaac M. Bavaresco]

Dear All,

I'm helping some fellow teachers with their courses of introductory
programming for control and automation.

They use Arduino, and the programming techniques they use make it very
difficult for the students to accomplish some goals.

In my opinion the problem resides in the traditional "setup/loop"
approach without additional advanced techniques for concurrency. The
programs become huge "spaghetti dishes" full of switch/case with
everything intermixed.

At some point it becomes impossible to understand, maintain or expand.

I'm creating a version of my co-routines framework for Arduino, which
allows a simple but powerful mechanism to write multiple independent
'tasks/co-routines' that run concurrently (it is a cooperative approach,
so each task must comply with some simple rules).

The zip file with version 0.1.2 is attached. I greatly appreciate any
comments and suggestions. Please check to example code.

What I'm looking for now is a meaningful example to show the full
potential of the co-routines. Something simple to understand but that
produces impressive results that would be hard to create using the
traditional approach.

Best regards,

Isaac




---
Este email foi escaneado pelo Avast antivírus.
https://www.avast.com/antivirus

[Jason White]

I don't have any advice about co-routines but I would like to offer the
following advice: sometimes using Threads and a RTOS can lead to easier to
understand code.

Blinking an LED with threading is easy: void BlinkLedThread() {
toggleLED(); Delay(1000) }
And at the same time there can be many other threads that read
push-buttons, drive LCDs, poll sensors, etc.

There is a learning curve; but for a teacher it would be pretty easy to
abstract away the RTOS so that the students can "just use it".

Regards,
-Jason White

On Sun, Jun 10, 2018 at 11:56 AM, Isaac M. Bavaresco <

Post by Isaac M. Bavaresco
Dear All,
I'm helping some fellow teachers with their courses of introductory
programming for control and automation.
They use Arduino, and the programming techniques they use make it very
difficult for the students to accomplish some goals.
In my opinion the problem resides in the traditional "setup/loop"
approach without additional advanced techniques for concurrency. The
programs become huge "spaghetti dishes" full of switch/case with
everything intermixed.
At some point it becomes impossible to understand, maintain or expand.
I'm creating a version of my co-routines framework for Arduino, which
allows a simple but powerful mechanism to write multiple independent
'tasks/co-routines' that run concurrently (it is a cooperative approach,
so each task must comply with some simple rules).
The zip file with version 0.1.2 is attached. I greatly appreciate any
comments and suggestions. Please check to example code.
What I'm looking for now is a meaningful example to show the full
potential of the co-routines. Something simple to understand but that
produces impressive results that would be hard to create using the
traditional approach.
Best regards,
Isaac
---
Este email foi escaneado pelo Avast antivírus.
https://www.avast.com/antivirus
--
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Isaac M. Bavaresco]

Jason,

A real RTOS is somewhat more complicated than co-routines and cannot be
implemented for every processor (PICs without software-accessible stack).

For the students that are straining to grasp the concepts of sequential
programming co-routines can be a good stepping stone before being
presented to an RTOS.

Cheers,

Isaac

Post by Jason White
I don't have any advice about co-routines but I would like to offer the
following advice: sometimes using Threads and a RTOS can lead to easier to
understand code.
Blinking an LED with threading is easy: void BlinkLedThread() {
toggleLED(); Delay(1000) }
And at the same time there can be many other threads that read
push-buttons, drive LCDs, poll sensors, etc.
There is a learning curve; but for a teacher it would be pretty easy to
abstract away the RTOS so that the students can "just use it".
Regards,
-Jason White
On Sun, Jun 10, 2018 at 11:56 AM, Isaac M. Bavaresco <

Post by Isaac M. Bavaresco
Dear All,
I'm helping some fellow teachers with their courses of introductory
programming for control and automation.
They use Arduino, and the programming techniques they use make it very
difficult for the students to accomplish some goals.
In my opinion the problem resides in the traditional "setup/loop"
approach without additional advanced techniques for concurrency. The
programs become huge "spaghetti dishes" full of switch/case with
everything intermixed.
At some point it becomes impossible to understand, maintain or expand.
I'm creating a version of my co-routines framework for Arduino, which
allows a simple but powerful mechanism to write multiple independent
'tasks/co-routines' that run concurrently (it is a cooperative approach,
so each task must comply with some simple rules).
The zip file with version 0.1.2 is attached. I greatly appreciate any
comments and suggestions. Please check to example code.
What I'm looking for now is a meaningful example to show the full
potential of the co-routines. Something simple to understand but that
produces impressive results that would be hard to create using the
traditional approach.
Best regards,
Isaac
---
Este email foi escaneado pelo Avast antivírus.
https://www.avast.com/antivirus
--
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Wouter van Ooijen]

Post by Isaac M. Bavaresco
I'm helping some fellow teachers with their courses of introductory
programming for control and automation.

Congratulations, you must be the 1000'th person who invented a
run-to-completion scheduler with sugaring that hides the RTC (google for
instance proto-threads).

The problem with this approach is that it can only sort-of hide the RTC
nature: thread-local information that must be preserved across a task
switch must be stored in globals, and task switching must happen in the
tasks's main function (there can be no waiting in a library call). IMO
this makes it more difficult for a student to understand what is happening.

We have a course that did use such an approach (RTC), but I replaced it
with a 'real' (but still cooperative, not preemptive) task switcher,
which made it much easier to explain and use.

[Isaac M. Bavaresco]

Post by Wouter van Ooijen
Congratulations, you must be the 1000'th person who invented a
run-to-completion scheduler with sugaring that hides the RTC (google for
instance proto-threads).

Wouter, I did not invent them. I was taught them about 30 years ago, by
the second or third person that invented them.

I know of proto-threads. FreeRTOS has co-routines too.

This framework was created by simplifying my RTOS code, changing it from
a full-blown preemptive RTOS to a simple single-stack scheduler.

My goal is just make co-routines more widespread known and used by those
students that are struggling to write programs just a little more
complicated than a line-follower.

Post by Wouter van Ooijen
The problem with this approach is that it can only sort-of hide the RTC
nature: thread-local information that must be preserved across a task
switch must be stored in globals, and task switching must happen in the
tasks's main function (there can be no waiting in a library call). IMO
this makes it more difficult for a student to understand what is happening.

Every approach has its own benefits and problems.

The spaghetti monolithic approach wastes no context-switching time and has virtually zero memory overhead.
The problem is that it is very hard to maintain and sometimes it becomes so complicated that the programmer doesn't understand anymore what is happening inside.

An RTOS simplifies a lot the timing management and allows a lot more modularity because, as you pointed it out, it can switch context inside subroutines,any level deep.
Its problems are the much greater use of memory (one stack for each task) and the great dependency of de RTOS's code with each platform. They are hard to port across architectures and some architectures can't even support them.
Another common issue is that it is much more prone to data corruption due to non-atomic access. It requires much more knowledge and care from the programmer.


In the middle ground, the co-routines solve a lot of the problems of the first approach and add very little memory and context-switching overhead.
They are easy to understand and are virtually free of non-atomic access problems.
The limitations do exist, but in my opinion it brings most of the benefits of the RTOS with just a small part of its drawbacks.

Post by Wouter van Ooijen
We have a course that did use such an approach (RTC), but I replaced it
with a 'real' (but still cooperative, not preemptive) task switcher,
which made it much easier to explain and use.

Do you teach them what is "under the hood"? Do they learn all the inner
workings of the task switcher?


Cheers,
Isaac


---
Este email foi escaneado pelo Avast antivírus.
https://www.avast.com/antivirus

[Isaac M. Bavaresco]

Kudo's for making a yield inside a subroutine possible!

Thanks!

But... does it nest, can you call a subroutine within a subroutine
within a subroutine and in that one do a yield?

Yes, you can nest subroutines that yield, but the time overhead may
become too high for more than two or three levels.

The other ugly part is the return type of the subroutine... Can your
technique handle other return types, or C++ user-defined operators?

The return type is just an integer in disguise. I use a custom typedef
so I can choose the most efficient type for each architecture without
having to change the application code. All that is necessary is changing
the typedef in the header file.
For 8-bit MCUs I prefer to use signed char, because it saves RAM, FLASH
and cycles.

Whenever possible I use negative values to signal error conditions, zero
for no error but no result and positive values for success conditions.
For instance, a routine to receive a serial packet. It would return a
negative value to signal an error (corrupted data, etc.), zero for
timeout (when timeout is not an error) and positive values matching the
number of bytes in the packet.

This is very efficient because errors can be detected by checking just
the MSBit (usually a single instruction in many platforms). If there is
no error then you have already the size of your packet in the return value.

Don't take me wrong, I don't say you did a bad job, but IMO this
run-to-completion with syntactic sugaring to hide the RTC-nature is
the wrong way to go. OTOH I am not sure the only alternative I know,
task switching, is the good way either. But at least it doesn't
po;llute the calling interface.

I agree that my solution is not the most elegant or clean solution, but
it is way cleaner and more organized than state-machine implementations.
Whenever possible I use an RTOS, but for some projects it is unfeasible.

But if you analyze the code you can see that it is easy to convert from
co-routine to RTOS and vice-versa. The flow is the same, it is just a
matter of removing the loops around the calls.


Cheers,
Isaac


---
Este email foi escaneado pelo Avast antivírus.
https://www.avast.com/antivirus

[RussellMc]

Hat: ANZAC "Just some guy, you know. Located semi-instantaneously in a
Canberra Oz hotel room, 1 Mbps internet. Home tomorrow". (Creased, tired
and moderately happy attire) - or similar.
*
__________

re



*​​> I'm helping some fellow teachers with their courses of introductory>
programming for control and automation.*
On 11 June 2018 at 05:36, Wouter van Ooijen <***@voti.nl> wrote:
Congratulations, you must be the 1000'th person who invented a

Post by Wouter van Ooijen
run-to-completion scheduler with sugaring that hides the RTC (google for
instance proto-threads).

​What have you done with Oli..., er, Wouter?

That's a remarkably acerbic response in the circumstances.
Odds are the figures is in the 10's or 100's of thousands, but, so what.

What YOU did is of interest and probably of value, and would probably be
even more so if you'd added or offered free access to your own solution in
a form suitable in this instance (Arduino, teacher/student, ...). No,
that's not a reasonably expected response from you (but would be marvellous
and welcome if it happened). But raining on someone else's parade, telling
us about your superior solution without and link or suggestions to a useful
path to it's implementation here seems less Wouterish than I'd have
expected.
[[Leaving the 1st paragraph or so off would have transformed your
response]].

FWIW - I was one of the 314,159 people who independently invented an
interupt / timer driven cooperative sort-of-RTOS - in my case probably
about 40 years ago (MC6800 initially :-) ). Despite it's obvious
limitations it's usefulness made it the obvious alternative to
no-RTOS-at-all in future systems of mine.

​

Post by Wouter van Ooijen
The problem with this approach is that it can only sort-of hide the RTC
nature: thread-local information that must be preserved across a task
switch must be stored in globals, and task switching must happen in the
tasks's main function (there can be no waiting in a library call). IMO
this makes it more difficult for a student to understand what is happening.
​Yes, mostly, and/but.

I have explained my approach to only a few people over the years, ​and each
would have been technically competent and well acquainted with
microcontrollers.
I may be wrong (it happens more often that I'd like :-) ) but I think I
could convey the information that was required "well enough" to any
diligent but newish programmer.
My approach works best in an environment where resources of RAM and time
are "not too too tightly constrained". [[ie it was not meant to be
performance optimised - extra magic was, in my world, supplied by me when
needed]].
- Cooperating tasks - which must, as you say, have sub-tasks which run to
completion each time they are 'run' (if the bit has arrived, process it,
otherwise go back to sleep // if the transmit buffer is not empty load a
byte, action
it, then go back to sleep // If the GPS half second clock ...)
- Subtasks must know how long they wish to sleep for before again being
woken.
- Task timers (or "alarm clock values" ) and, possibly, related "do it now"
flags are processed by a magic servant with feet in the IRQ and background
worlds.

​Again, my system was not made to be "explicable" but, I think, would be
able to be formalised enough to be conveyed to diligent students.
For the type of Arduino user that Olin complains of, the whole system tends
to be somewhat mysterious and and add ons the more so. ​
​But they seem unlikely to be taking the course described. (I may be wrong
:-) ).

We have a course that did use such an approach (RTC), but I replaced it

Post by Wouter van Ooijen
with a 'real' (but still cooperative, not preemptive) task switcher,
which made it much easier to explain and use.
​Better is usually better, if it can be realistically achieved.

​PS: Keep up the good work. ​


​Russell

*
https://www.google.co.nz/search?biw=1366&bih=635&tbm=isch&sa=1&ei=WkUeW-WwO4ic8QWfp73ACQ&q=anzac+hat&oq=anzac+hat&gs_l=img.3..0l6j0i8i30k1l4.3498.6575.0.7154.6.5.1.0.0.0.264.1045.0j2j3.5.0....0...1c.1.64.img..0.6.1049...0i67k1j0i24k1j0i10i24k1.0.LXaR_P90kmA
​

[Wouter van Ooijen]

Post by RussellMc
What YOU did is of interest and probably of value, and would probably be
even more so if you'd added or offered free access to your own solution in
a form suitable in this instance (Arduino, teacher/student, ...). No,
that's not a reasonably expected response from you (but would be marvellous
and welcome if it happened). But raining on someone else's parade, telling
us about your superior solution without and link or suggestions to a useful
path to it's implementation here seems less Wouterish than I'd have
expected.

It was one of my regular rants against the superiority ( / useability) of State Machines and Run-To-Completion systems, either plain or disguised in some
syntactic sugar.

I don't claim my solution is superior in all aspects, but in general I prefer context switching. But the achilles heel is the multitude of stacks. For a simple task like regularly copying one GPIO to another a RTC/FSM approach would need a few FLASH instructions, but (on a Cortex M) a task switcher would need at least 80 bytes (of the more scarce RAM).

OTOH a RTC/FSM appeoach pollutes the API: an API for the simple task of reading a pin value needs to show whether this is done without waiting (simple function call to read a local GPIO) or with waiting (I2C access to a pin on a GPIO extender).

But my stuff is freely available (but maybe not easily useable without me holding your hand - that's not your fault but mine for not documenting everything properly!):

   www.github.com/wovo/rtos

You would probably need hwlib and bmptk to get it working.

[Dwayne Reid]

Hi there, Isaac.

No attachment received at my computer.

Many thanks!

dwayne

Post by Isaac M. Bavaresco
Dear All,
I'm helping some fellow teachers with their courses of introductory
programming for control and automation.
They use Arduino, and the programming techniques they use make it very
difficult for the students to accomplish some goals.
In my opinion the problem resides in the traditional "setup/loop"
approach without additional advanced techniques for concurrency. The
programs become huge "spaghetti dishes" full of switch/case with
everything intermixed.
At some point it becomes impossible to understand, maintain or expand.
I'm creating a version of my co-routines framework for Arduino, which
allows a simple but powerful mechanism to write multiple independent
'tasks/co-routines' that run concurrently (it is a cooperative approach,
so each task must comply with some simple rules).
The zip file with version 0.1.2 is attached. I greatly appreciate any
comments and suggestions. Please check to example code.
What I'm looking for now is a meaningful example to show the full
potential of the co-routines. Something simple to understand but that
produces impressive results that would be hard to create using the
traditional approach.
Best regards,
Isaac

--
Dwayne Reid <***@planet.eon.net>
Trinity Electronics Systems Ltd Edmonton, AB, CANADA
780-489-3199 voice 780-487-6397 fax [phone] Toll Free
www.trinity-electronics.com
Custom Electronics Design and Manufacturing

[Dwayne Reid]

Scratch what I just wrote - I must be having one of "THOSE" days <grin>.

Attachment is indeed present here.

Sorry.

dwayne


Hi there, Isaac.

No attachment received at my computer.

Many thanks!

dwayne

Post by Isaac M. Bavaresco
Dear All,
I'm helping some fellow teachers with their courses of introductory
programming for control and automation.
They use Arduino, and the programming techniques they use make it very
difficult for the students to accomplish some goals.
In my opinion the problem resides in the traditional "setup/loop"
approach without additional advanced techniques for concurrency. The
programs become huge "spaghetti dishes" full of switch/case with
everything intermixed.
At some point it becomes impossible to understand, maintain or expand.
I'm creating a version of my co-routines framework for Arduino, which
allows a simple but powerful mechanism to write multiple independent
'tasks/co-routines' that run concurrently (it is a cooperative approach,
so each task must comply with some simple rules).
The zip file with version 0.1.2 is attached. I greatly appreciate any
comments and suggestions. Please check to example code.
What I'm looking for now is a meaningful example to show the full
potential of the co-routines. Something simple to understand but that
produces impressive results that would be hard to create using the
traditional approach.
Best regards,
Isaac

--
Dwayne Reid <***@planet.eon.net>
Trinity Electronics Systems Ltd Edmonton, AB, CANADA
780-489-3199 voice 780-487-6397 fax [phone] Toll Free
www.trinity-electronics.com
Custom Electronics Design and Manufacturing

[Isaac M. Bavaresco]

Just now I noticed that I was replying directly to Wouter.

Forwarding so the others can follow the thread.



-------- Mensagem encaminhada --------
Assunto: Re: CoRoutines library for Arduino
Data: Sun, 10 Jun 2018 19:04:25 -0300

Post by Isaac M. Bavaresco
Every approach has its own benefits and problems.

True. I have a friend who swears by the RTC style of programming.

Now I swear by the state-machine style, although I used it a lot and
have some old code that still use them.

IMO the list-of-state-machines approach is not that difficult to explain.

It is indeed very simple to understand.
The problem arises when the state machines become large and with time
very complicated. Adding states or changing the logic sometimes require
a lot of code rewriting.
Analyzing old projects I found that frequently I did things less
efficiently than they could have been because dealing with state/case
management distracted me from the problem at hand.
I have old projects that grew over time and I can't understand the
state-machines anymore. I'm sure that although they work, some parts
doesn't behave exactly how I think they should.

Porting is limited to the stack switching (10-20 lines of assembler)
and setting up the initial stack (5 lines of C/C++).

For what architecture do you use it? Can you share the code?

Are there any architectures left without an acessible stack except the
PIC 12 and 14 bit cores?

I don't think there are many that matter anyway.
But there are a lot of small 8-bit MCUs with too little RAM to be
practical to use an RTOS.
For PIC12, PIC14 and some PIC18 I use co-routines if any level of
concurrency is required.

Post by Isaac M. Bavaresco
Another common issue is that it is much more prone to data corruption
due to non-atomic access. It requires much more knowledge and care
from the programmer.

That's for preemptive switching, which offers some serious advantages
(faster response time) but as you say opens up a few cans of worms. I
perfer cooperative multi-threading.

When I started writing my first RTOS I found that the distance between
non-preemptive and preemptive was so small that I decided to take the
longer path.

Cheers,
Isaac



---
Este email foi escaneado pelo Avast antivírus.
https://www.avast.com/antivirus

[Isaac M. Bavaresco]

-------- Mensagem encaminhada --------
Assunto: Re: CoRoutines library for Arduino
Data: Sun, 10 Jun 2018 23:02:02 -0300
De: Isaac M. Bavaresco <***@gmail.com>
Para: Wouter van Ooijen <***@voti.nl>



Wouter,

See the attached file. It shows programming-patterns for some frequent
programming situations with co-routines.

It is pseudo-code with pieces picked here and there and is missing some
parts, don't expect it to work.

Cheers,

Isaac

Post by Isaac M. Bavaresco
Every approach has its own benefits and problems.

True. I have a friend who swears by the RTC style of programming.

Post by Isaac M. Bavaresco
The spaghetti monolithic approach wastes no context-switching time
and has virtually zero memory overhead.
The problem is that it is very hard to maintain and sometimes it
becomes so complicated that the programmer doesn't understand anymore
what is happening inside.

IMO the list-of-state-machines approach is not that difficult to explain.

Post by Isaac M. Bavaresco
An RTOS simplifies a lot the timing management and allows a lot more
modularity because, as you pointed it out, it can switch context
inside subroutines,any level deep.
Its problems are the much greater use of memory (one stack for each
task) and the great dependency of de RTOS's code with each platform.
They are hard to port across architectures and some architectures
can't even support them.

Porting is limited to the stack switching (10-20 lines of assembler)
and setting up the initial stack (5 lines of C/C++).
Are there any architectures left without an acessible stack except the
PIC 12 and 14 bit cores?

Post by Isaac M. Bavaresco
Another common issue is that it is much more prone to data corruption
due to non-atomic access. It requires much more knowledge and care
from the programmer.

That's for preemptive switching, which offers some serious advantages
(faster response time) but as you say opens up a few cans of worms. I
perfer cooperative multi-threading.

Post by Isaac M. Bavaresco
In the middle ground, the co-routines solve a lot of the problems of
the first approach and add very little memory and context-switching
overhead.

The memory overhead is indeed the weak point of threading, especially
when guess must be made for the stack size.

Post by Isaac M. Bavaresco
They are easy to understand and are virtually free of non-atomic
access problems.
The limitations do exist, but in my opinion it brings most of the
benefits of the RTOS with just a small part of its drawbacks.

Post by Wouter van Ooijen
We have a course that did use such an approach (RTC), but I replaced it
with a 'real' (but still cooperative, not preemptive) task switcher,
which made it much easier to explain and use.

Do you teach them what is "under the hood"?

The task switching part: yes. The course is mixed C++/assembler.

Post by Isaac M. Bavaresco
Do they learn all the inner
workings of the task switcher?

Not everything, the synchronization primitives (like fixed-max-length
queues) are just library elements. But they don't contain any magic
that the students couldn't understand.

---
Este email foi escaneado pelo Avast antivírus.
https://www.avast.com/antivirus