[PIC] Sum of digits

[Brent Brown]

Hi all,

I have a math problem that is a little interesting. Background first: I'm storing
"operating hours" in internal EEPROM on a PIC16F887. It's in a product I've been
making for some time, but despite my best efforts so far the hours counter
occasionally gets corrupted. I suspect the biggest problem is writing to EEPROM in
a "dying gasp" when power supply falls, with simply not enough time to reliably do
so.

A new scheme is planned. Hours increment in units 0.01hr (36s) up to 9999.99,
then roll over to 0. On each increment it will be stored in a different EEPROM
location. This addresses three issues 1) the byte endurance of EEPROM (1M typ
but 100k guaranteed), 2) ability to retrieve a previous record if corruption is
detected, loosing as little as 36 seconds (fairly insignificant), and 3) no longer
writing to EEPROM as power fails.

I plan to encode the number essentially as 6 x BCD digits, 1 digit per byte. I take the
BCD value, add 3 to it (avoids 0 being a valid code), shift to upper nibble, and write
the inverted value into the lower nibble. This results in 10 symbols for numbers 0-9
as follows: 0x3C, 0x4B, 0x5A, 0x69, 0x78, 0x87, 0x96, 0xA5, 0xB4, 0xC3.

Corruption in a digit is detected when a byte does not equate to one of the valid
symbols (10 valid 8 bit codes out of 256). In addition I'll add a checksum. Preferably
I'd like the checksum to never equal FF, as that is the "erased" value of EEPROM, I
will also use that as a "start of record" marker. Thus a complete record will be 8
bytes:

start_record, digit1, digit2, digit3, digit4, digit5, digit6, checksum

Example: (0000.01 hours)

0xFF, 0x3C, 0x3C, 0x3C, 0x3C, 0x3C, 0x4B, 0x77

My curious math question is can the checksum ever be 0xFF? It's complicated by
the values I'm using for the symbols, and the checksum being the LS 8 bits of the
sum.

Simplifying the symbols back to decimal values the question becomes: in the
sequence 000000 - 999999 what are all the possible values for the sum of digits?
One can see fairly easily that the minimum would be 0+0+0+0+0+0=0, the
maximum would be 9+9+9+9+9+9=54. Also: there are 1 million possible sums
(10^6), all will be in the range 0-54 (9*6), all integer values will be covered as the
step size is 1, and there will be many identical results.

Wondering if/how I could solve this mathematically. I did make a spreadsheet with
1000 rows to calculate the sums of digits from 000-999 and graphed the results in a
histogram. 1,000,000 rows is possible in Excel, but takes a bit of time, working on
that now. Interestingly, the 1000 row spreadsheet showed sum of digits covered all
values from 0-27, as expected, and the distribution of values is a bell curve.

I can of course work around and allow for the checksum being 0xFF, or change the
value if/when it ever does match 0xFF, but still curious to know if it is necessary.


--
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Neil]

Here's a slightly alternate option, which I use for odometer data. That
the data is always increasing is required for this to work.

I virtually divide the EEPROM space into groups of 5 bytes. 4 for
storing the data, and one as a "dirty" flag.
There's a special value indicating that the group is clean/good. When I
write a new value, I first set the dirty byte to anything other than
that special value, write all the bytes to hold the new value, verify
everything, then set the dirty bit to the special value. Then I verify
or repeat to ensure that value gets saved.
On power up, I scan through all groups, and for all clean groups, I pick
the highest value, then start writing new values in the group after that.

I store a new value every 0.01 miles, so theoretically/statistically I
could be losing 0.005 miles on average for every power cycle. I could
just re-add that on each power-up, but haven't (yet?).

This has worked well for over a decade now, and I used it for a Hobbs
hour meter with no issues reported also.

You'll need to think about what happens if you have a roll-over, but at
first thought you can clear all the data (mark all the dirty bytes as
dirty), when you do so.

Cheers,
-Neil.



On 12/10/2017 7:52 PM, Brent Brown wrote:
> Hi all,
>
> I have a math problem that is a little interesting. Background first: I'm storing
> "operating hours" in internal EEPROM on a PIC16F887. It's in a product I've been
> making for some time, but despite my best efforts so far the hours counter
> occasionally gets corrupted. I suspect the biggest problem is writing to EEPROM in
> a "dying gasp" when power supply falls, with simply not enough time to reliably do
> so.
>
> A new scheme is planned. Hours increment in units 0.01hr (36s) up to 9999.99,
> then roll over to 0. On each increment it will be stored in a different EEPROM
> location. This addresses three issues 1) the byte endurance of EEPROM (1M typ
> but 100k guaranteed), 2) ability to retrieve a previous record if corruption is
> detected, loosing as little as 36 seconds (fairly insignificant), and 3) no longer
> writing to EEPROM as power fails.
>
> I plan to encode the number essentially as 6 x BCD digits, 1 digit per byte. I take the
> BCD value, add 3 to it (avoids 0 being a valid code), shift to upper nibble, and write
> the inverted value into the lower nibble. This results in 10 symbols for numbers 0-9
> as follows: 0x3C, 0x4B, 0x5A, 0x69, 0x78, 0x87, 0x96, 0xA5, 0xB4, 0xC3.
>
> Corruption in a digit is detected when a byte does not equate to one of the valid
> symbols (10 valid 8 bit codes out of 256). In addition I'll add a checksum. Preferably
> I'd like the checksum to never equal FF, as that is the "erased" value of EEPROM, I
> will also use that as a "start of record" marker. Thus a complete record will be 8
> bytes:
>
> start_record, digit1, digit2, digit3, digit4, digit5, digit6, checksum
>
> Example: (0000.01 hours)
>
> 0xFF, 0x3C, 0x3C, 0x3C, 0x3C, 0x3C, 0x4B, 0x77
>
> My curious math question is can the checksum ever be 0xFF? It's complicated by
> the values I'm using for the symbols, and the checksum being the LS 8 bits of the
> sum.
>
> Simplifying the symbols back to decimal values the question becomes: in the
> sequence 000000 - 999999 what are all the possible values for the sum of digits?
> One can see fairly easily that the minimum would be 0+0+0+0+0+0=0, the
> maximum would be 9+9+9+9+9+9=54. Also: there are 1 million possible sums
> (10^6), all will be in the range 0-54 (9*6), all integer values will be covered as the
> step size is 1, and there will be many identical results.
>
> Wondering if/how I could solve this mathematically. I did make a spreadsheet with
> 1000 rows to calculate the sums of digits from 000-999 and graphed the results in a
> histogram. 1,000,000 rows is possible in Excel, but takes a bit of time, working on
> that now. Interestingly, the 1000 row spreadsheet showed sum of digits covered all
> values from 0-27, as expected, and the distribution of values is a bell curve.
>
> I can of course work around and allow for the checksum being 0xFF, or change the
> value if/when it ever does match 0xFF, but still curious to know if it is necessary.
>
>

--
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Brent Brown]

Hi Neil,

Does sound very similar to what I'm proposing to do, and encouraging that it's been
working out well for you. I take your point of the careful sequnce in making sure the
data is written/validated/marked clean/re-validated.

And I like the idea of the clean/dirty flag. Previously I wasn't planning on having one,
but see it could be helpful in managing the case where hours rollover from 9999.99
to 0, when it's no longer as simple as just checking for the record with the highest
count.

Previously I was recording 1000ths of an hour to EERPOM whenever there was a
power down event (12V supply drops below 10V = expected power failure iminent)
and regular EEPROM updates every 0.1 operating hours. On power up I would
pickup the 1000ths and continue from there so nothing was lost. I may do so again,
but I'm still a little nervous about writing while power is starting to fail, since I'm not
100% sure where my odd corruption problems have been coming from.

Thanks for the tips~!

On 10 Dec 2017 at 22:42, Neil wrote:

> Here's a slightly alternate option, which I use for odometer data. That
> the data is always increasing is required for this to work.
>
> I virtually divide the EEPROM space into groups of 5 bytes. 4 for
> storing the data, and one as a "dirty" flag.
> There's a special value indicating that the group is clean/good. When I
> write a new value, I first set the dirty byte to anything other than
> that special value, write all the bytes to hold the new value, verify
> everything, then set the dirty bit to the special value. Then I verify
> or repeat to ensure that value gets saved.
> On power up, I scan through all groups, and for all clean groups, I pick
> the highest value, then start writing new values in the group after that.
>
> I store a new value every 0.01 miles, so theoretically/statistically I
> could be losing 0.005 miles on average for every power cycle. I could
> just re-add that on each power-up, but haven't (yet?).
>
> This has worked well for over a decade now, and I used it for a Hobbs
> hour meter with no issues reported also.
>
> You'll need to think about what happens if you have a roll-over, but at
> first thought you can clear all the data (mark all the dirty bytes as
> dirty), when you do so.
>
> Cheers,
> -Neil.
>
>
>
> On 12/10/2017 7:52 PM, Brent Brown wrote:
> > Hi all,
> >
> > I have a math problem that is a little interesting. Background first: I'm storing
> > "operating hours" in internal EEPROM on a PIC16F887. It's in a product I've been
> > making for some time, but despite my best efforts so far the hours counter
> > occasionally gets corrupted. I suspect the biggest problem is writing to EEPROM in
> > a "dying gasp" when power supply falls, with simply not enough time to reliably do
> > so.
> >
> > A new scheme is planned. Hours increment in units 0.01hr (36s) up to 9999.99,
> > then roll over to 0. On each increment it will be stored in a different EEPROM
> > location. This addresses three issues 1) the byte endurance of EEPROM (1M typ
> > but 100k guaranteed), 2) ability to retrieve a previous record if corruption is
> > detected, loosing as little as 36 seconds (fairly insignificant), and 3) no longer
> > writing to EEPROM as power fails.
> >
> > I plan to encode the number essentially as 6 x BCD digits, 1 digit per byte. I take the
> > BCD value, add 3 to it (avoids 0 being a valid code), shift to upper nibble, and write
> > the inverted value into the lower nibble. This results in 10 symbols for numbers 0-9
> > as follows: 0x3C, 0x4B, 0x5A, 0x69, 0x78, 0x87, 0x96, 0xA5, 0xB4, 0xC3.
> >
> > Corruption in a digit is detected when a byte does not equate to one of the valid
> > symbols (10 valid 8 bit codes out of 256). In addition I'll add a checksum. Preferably
> > I'd like the checksum to never equal FF, as that is the "erased" value of EEPROM, I
> > will also use that as a "start of record" marker. Thus a complete record will be 8
> > bytes:
> >
> > start_record, digit1, digit2, digit3, digit4, digit5, digit6, checksum
> >
> > Example: (0000.01 hours)
> >
> > 0xFF, 0x3C, 0x3C, 0x3C, 0x3C, 0x3C, 0x4B, 0x77
> >
> > My curious math question is can the checksum ever be 0xFF? It's complicated by
> > the values I'm using for the symbols, and the checksum being the LS 8 bits of the
> > sum.
> >
> > Simplifying the symbols back to decimal values the question becomes: in the
> > sequence 000000 - 999999 what are all the possible values for the sum of digits?
> > One can see fairly easily that the minimum would be 0+0+0+0+0+0=0, the
> > maximum would be 9+9+9+9+9+9=54. Also: there are 1 million possible sums
> > (10^6), all will be in the range 0-54 (9*6), all integer values will be covered as the
> > step size is 1, and there will be many identical results.
> >
> > Wondering if/how I could solve this mathematically. I did make a spreadsheet with
> > 1000 rows to calculate the sums of digits from 000-999 and graphed the results in a
> > histogram. 1,000,000 rows is possible in Excel, but takes a bit of time, working on
> > that now. Interestingly, the 1000 row spreadsheet showed sum of digits covered all
> > values from 0-27, as expected, and the distribution of values is a bell curve.
> >
> > I can of course work around and allow for the checksum being 0xFF, or change the
> > value if/when it ever does match 0xFF, but still curious to know if it is necessary.
> >
> >
>
> --
> http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
> View/change your membership options at
> http://mailman.mit.edu/mailman/listinfo/piclist


--
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Jason White]

Hello Brent,

I'm no mathematician and I suspect an analytical solution would be
nontrivial due to the modulo.

However the problem is easily solved by writing a C program
(attached). I have written a program that computes the checksum for
every possible digit combination and writes the number of occurrences
for each checksum into a table. I have also attached the result tables
for the set of digits 0-9 (used to test the validity of the program)
and for your set of 10 BCD-like digits.

It would appear that the answer to your question is: no. The checksum
0xFF does not appear. Although many other numbers do show up.

-Jason

Below is a list of all checksums that appear a nonzero number of times.

Checksum 0x0b: 2002
Checksum 0x0c: 54747
Checksum 0x0d: 4332
Checksum 0x1a: 1287
Checksum 0x1b: 53262
Checksum 0x1c: 6062
Checksum 0x29: 792
Checksum 0x2a: 50877
Checksum 0x2b: 8232
Checksum 0x38: 462
Checksum 0x39: 47712
Checksum 0x3a: 10872
Checksum 0x47: 252
Checksum 0x48: 43917
Checksum 0x49: 13992
Checksum 0x56: 126
Checksum 0x57: 39662
Checksum 0x58: 17577
Checksum 0x65: 56
Checksum 0x66: 35127
Checksum 0x67: 21582
Checksum 0x68: 1
Checksum 0x74: 21
Checksum 0x75: 30492
Checksum 0x76: 25927
Checksum 0x77: 6
Checksum 0x83: 6
Checksum 0x84: 25927
Checksum 0x85: 30492
Checksum 0x86: 21
Checksum 0x92: 1
Checksum 0x93: 21582
Checksum 0x94: 35127
Checksum 0x95: 56
Checksum 0xa2: 17577
Checksum 0xa3: 39662
Checksum 0xa4: 126
Checksum 0xb1: 13992
Checksum 0xb2: 43917
Checksum 0xb3: 252
Checksum 0xc0: 10872
Checksum 0xc1: 47712
Checksum 0xc2: 462
Checksum 0xcf: 8232
Checksum 0xd0: 50877
Checksum 0xd1: 792
Checksum 0xde: 6062
Checksum 0xdf: 53262
Checksum 0xe0: 1287
Checksum 0xed: 4332
Checksum 0xee: 54747
Checksum 0xef: 2002
Checksum 0xfc: 2997
Checksum 0xfd: 55252
Checksum 0xfe: 2997

[Brent Brown]

Jason,

Much appreciated, you've almost written more code to test this than I will have in
the application~!

I persisted with Excel and produced a list of 1M results and did a sorted count by
checksum value. Side by side it agrees with your results perfectly~! Excel is a little
slow when it's working with 10^6 rows, copy and paste keeps it busy for 10-20s,
then about 5s to recalculate the whole sheet... so it definitely took a little longer than
your C programs 1.8s.

Yes, the modulo 256 makes the results "wrap around" in some weird places, but the
good news is 0xFF is not in the set of results so I can happily use that.

On 10 Dec 2017 at 23:28, Jason White wrote:

> Hello Brent,
>
> I'm no mathematician and I suspect an analytical solution would be
> nontrivial due to the modulo.
>
> However the problem is easily solved by writing a C program
> (attached). I have written a program that computes the checksum for
> every possible digit combination and writes the number of occurrences
> for each checksum into a table. I have also attached the result tables
> for the set of digits 0-9 (used to test the validity of the program)
> and for your set of 10 BCD-like digits.
>
> It would appear that the answer to your question is: no. The checksum
> 0xFF does not appear. Although many other numbers do show up.
>
> -Jason
>
> Below is a list of all checksums that appear a nonzero number of times.
>
> Checksum 0x0b: 2002
> Checksum 0x0c: 54747
> Checksum 0x0d: 4332
> Checksum 0x1a: 1287
> Checksum 0x1b: 53262
> Checksum 0x1c: 6062
> Checksum 0x29: 792
> Checksum 0x2a: 50877
> Checksum 0x2b: 8232
> Checksum 0x38: 462
> Checksum 0x39: 47712
> Checksum 0x3a: 10872
> Checksum 0x47: 252
> Checksum 0x48: 43917
> Checksum 0x49: 13992
> Checksum 0x56: 126
> Checksum 0x57: 39662
> Checksum 0x58: 17577
> Checksum 0x65: 56
> Checksum 0x66: 35127
> Checksum 0x67: 21582
> Checksum 0x68: 1
> Checksum 0x74: 21
> Checksum 0x75: 30492
> Checksum 0x76: 25927
> Checksum 0x77: 6
> Checksum 0x83: 6
> Checksum 0x84: 25927
> Checksum 0x85: 30492
> Checksum 0x86: 21
> Checksum 0x92: 1
> Checksum 0x93: 21582
> Checksum 0x94: 35127
> Checksum 0x95: 56
> Checksum 0xa2: 17577
> Checksum 0xa3: 39662
> Checksum 0xa4: 126
> Checksum 0xb1: 13992
> Checksum 0xb2: 43917
> Checksum 0xb3: 252
> Checksum 0xc0: 10872
> Checksum 0xc1: 47712
> Checksum 0xc2: 462
> Checksum 0xcf: 8232
> Checksum 0xd0: 50877
> Checksum 0xd1: 792
> Checksum 0xde: 6062
> Checksum 0xdf: 53262
> Checksum 0xe0: 1287
> Checksum 0xed: 4332
> Checksum 0xee: 54747
> Checksum 0xef: 2002
> Checksum 0xfc: 2997
> Checksum 0xfd: 55252
> Checksum 0xfe: 2997
>


--
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Isaac M. Bavaresco]

Those checksums seem to be very biased and probably are not good for
ensuring data integrity.

Why not using CRC? CRC8 is probably much stronger than any checksum and
easy to implement. It is computationally light too.

A reference implementation can be found in Maxim-Dalla's 1Wire protocol.


Cheers,

Isaac



Em 11/12/2017 02:28, Jason White escreveu:
> Hello Brent,
>
> I'm no mathematician and I suspect an analytical solution would be
> nontrivial due to the modulo.
>
> However the problem is easily solved by writing a C program
> (attached). I have written a program that computes the checksum for
> every possible digit combination and writes the number of occurrences
> for each checksum into a table. I have also attached the result tables
> for the set of digits 0-9 (used to test the validity of the program)
> and for your set of 10 BCD-like digits.
>
> It would appear that the answer to your question is: no. The checksum
> 0xFF does not appear. Although many other numbers do show up.
>
> -Jason
>
> Below is a list of all checksums that appear a nonzero number of times.
>
> Checksum 0x0b: 2002
> Checksum 0x0c: 54747
> Checksum 0x0d: 4332
> Checksum 0x1a: 1287
> Checksum 0x1b: 53262
> Checksum 0x1c: 6062
> Checksum 0x29: 792
> Checksum 0x2a: 50877
> Checksum 0x2b: 8232
> Checksum 0x38: 462
> Checksum 0x39: 47712
> Checksum 0x3a: 10872
> Checksum 0x47: 252
> Checksum 0x48: 43917
> Checksum 0x49: 13992
> Checksum 0x56: 126
> Checksum 0x57: 39662
> Checksum 0x58: 17577
> Checksum 0x65: 56
> Checksum 0x66: 35127
> Checksum 0x67: 21582
> Checksum 0x68: 1
> Checksum 0x74: 21
> Checksum 0x75: 30492
> Checksum 0x76: 25927
> Checksum 0x77: 6
> Checksum 0x83: 6
> Checksum 0x84: 25927
> Checksum 0x85: 30492
> Checksum 0x86: 21
> Checksum 0x92: 1
> Checksum 0x93: 21582
> Checksum 0x94: 35127
> Checksum 0x95: 56
> Checksum 0xa2: 17577
> Checksum 0xa3: 39662
> Checksum 0xa4: 126
> Checksum 0xb1: 13992
> Checksum 0xb2: 43917
> Checksum 0xb3: 252
> Checksum 0xc0: 10872
> Checksum 0xc1: 47712
> Checksum 0xc2: 462
> Checksum 0xcf: 8232
> Checksum 0xd0: 50877
> Checksum 0xd1: 792
> Checksum 0xde: 6062
> Checksum 0xdf: 53262
> Checksum 0xe0: 1287
> Checksum 0xed: 4332
> Checksum 0xee: 54747
> Checksum 0xef: 2002
> Checksum 0xfc: 2997
> Checksum 0xfd: 55252
> Checksum 0xfe: 2997
>
>



---
Este email foi escaneado pelo Avast antivírus.
https://www.avast.com/antivirus
--
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
View/change your membership options at
http://

[Brent Brown]

On 11 Dec 2017 at 21:51, Isaac M. Bavaresco wrote:

> Those checksums seem to be very biased and probably are not good for
> ensuring data integrity.
>
> Why not using CRC? CRC8 is probably much stronger than any checksum and
> easy to implement. It is computationally light too.

Right you are Isaac.

There was some effort required in getting the CRC function implemented correctly,
(basically choosing from existing code examples, then testing to make sure it works
as expected), but the performance gain looks really impressive.

Found this CRC code I had used before, taken from an example in an RFID tag
reader manual, apparently CCITT-CRC-8 polygon x^8+x^4+x^3+x^2+1.

char CalcCRC8(char CRC, char byte){
unsigned char count;
for (count = 0; count < 8; ++count){
if (((CRC & 0x01) ^ (byte & 0x01)) != 0){
CRC ^= 0x70;
CRC >>= 1;
CRC |= 0x80;
}
else{
CRC >>= 1;
CRC &= 0x7F;
}
byte >>= 1;
}
return CRC;
}

>From the C code Jason sent through (thanks again) I modified it to produce results
for the basic 8 bit checksum and the 8 bit CRC function above. To recap, there are
6 digits of 10 symbols (0x3C, 0x4B, 0x5A, 0x69, 0x78, 0x87, 0x96, 0xA5, 0xB4,
0xC3), representing all 10^6 (1 million) possible codes of decimal numbers from
000000 - 999999.

Below is a list of all the 8 bit checksum/CRC values in Hex (first coulmn, 256 of
them), then the number of times (decimal) each appears for a simple checksum
(second column) and CRC (third column). The performance of CRC is exceptional...
all 256 possible results are represented, with near perfect distribution (1/256 x
1000000 = 3906).

For my previously mentioned requirement of the check value not equalling 0xFF, I
think I can live with testing for and replacing that occurence with a different code.

0x00: 0 3906
0x01: 0 3911
0x02: 0 3906
0x03: 0 3911
0x04: 0 3911
0x05: 0 3906
0x06: 0 3911
0x07: 0 3906
0x08: 0 3904
0x09: 0 3904
0x0A: 0 3904
0x0B: 2002 3904
0x0C: 54747 3904
0x0D: 4332 3904
0x0E: 0 3904
0x0F: 0 3904
0x10: 0 3906
0x11: 0 3911
0x12: 0 3906
0x13: 0 3911
0x14: 0 3911
0x15: 0 3906
0x16: 0 3911
0x17: 0 3906
0x18: 0 3904
0x19: 0 3904
0x1A: 1287 3904
0x1B: 53262 3904
0x1C: 6062 3904
0x1D: 0 3904
0x1E: 0 3904
0x1F: 0 3904
0x20: 0 3911
0x21: 0 3906
0x22: 0 3911
0x23: 0 3906
0x24: 0 3906
0x25: 0 3911
0x26: 0 3906
0x27: 0 3911
0x28: 0 3904
0x29: 792 3904
0x2A: 50877 3904
0x2B: 8232 3904
0x2C: 0 3904
0x2D: 0 3904
0x2E: 0 3904
0x2F: 0 3904
0x30: 0 3911
0x31: 0 3906
0x32: 0 3911
0x33: 0 3906
0x34: 0 3906
0x35: 0 3911
0x36: 0 3906
0x37: 0 3911
0x38: 462 3904
0x39: 47712 3904
0x3A: 10872 3904
0x3B: 0 3904
0x3C: 0 3904
0x3D: 0 3904
0x3E: 0 3904
0x3F: 0 3904
0x40: 0 3911
0x41: 0 3906
0x42: 0 3911
0x43: 0 3906
0x44: 0 3906
0x45: 0 3911
0x46: 0 3906
0x47: 252 3911
0x48: 43917 3904
0x49: 13992 3904
0x4A: 0 3904
0x4B: 0 3904
0x4C: 0 3904
0x4D: 0 3904
0x4E: 0 3904
0x4F: 0 3904
0x50: 0 3911
0x51: 0 3906
0x52: 0 3911
0x53: 0 3906
0x54: 0 3906
0x55: 0 3911
0x56: 126 3906
0x57: 39662 3911
0x58: 17577 3904
0x59: 0 3904
0x5A: 0 3904
0x5B: 0 3904
0x5C: 0 3904
0x5D: 0 3904
0x5E: 0 3904
0x5F: 0 3904
0x60: 0 3906
0x61: 0 3911
0x62: 0 3906
0x63: 0 3911
0x64: 0 3911
0x65: 56 3906
0x66: 35127 3911
0x67: 21582 3906
0x68: 1 3904
0x69: 0 3904
0x6A: 0 3904
0x6B: 0 3904
0x6C: 0 3904
0x6D: 0 3904
0x6E: 0 3904
0x6F: 0 3904
0x70: 0 3906
0x71: 0 3911
0x72: 0 3906
0x73: 0 3911
0x74: 21 3911
0x75: 30492 3906
0x76: 25927 3911
0x77: 6 3906
0x78: 0 3904
0x79: 0 3904
0x7A: 0 3904
0x7B: 0 3904
0x7C: 0 3904
0x7D: 0 3904
0x7E: 0 3904
0x7F: 0 3904
0x80: 0 3906
0x81: 0 3911
0x82: 0 3906
0x83: 6 3911
0x84: 25927 3911
0x85: 30492 3906
0x86: 21 3911
0x87: 0 3906
0x88: 0 3904
0x89: 0 3904
0x8A: 0 3904
0x8B: 0 3904
0x8C: 0 3904
0x8D: 0 3904
0x8E: 0 3904
0x8F: 0 3904
0x90: 0 3906
0x91: 0 3911
0x92: 1 3906
0x93: 21582 3911
0x94: 35127 3911
0x95: 56 3906
0x96: 0 3911
0x97: 0 3906
0x98: 0 3904
0x99: 0 3904
0x9A: 0 3904
0x9B: 0 3904
0x9C: 0 3904
0x9D: 0 3904
0x9E: 0 3904
0x9F: 0 3904
0xA0: 0 3911
0xA1: 0 3906
0xA2: 17577 3911
0xA3: 39662 3906
0xA4: 126 3906
0xA5: 0 3911
0xA6: 0 3906
0xA7: 0 3911
0xA8: 0 3904
0xA9: 0 3904
0xAA: 0 3904
0xAB: 0 3904
0xAC: 0 3904
0xAD: 0 3904
0xAE: 0 3904
0xAF: 0 3904
0xB0: 0 3911
0xB1: 13992 3906
0xB2: 43917 3911
0xB3: 252 3906
0xB4: 0 3906
0xB5: 0 3911
0xB6: 0 3906
0xB7: 0 3911
0xB8: 0 3904
0xB9: 0 3904
0xBA: 0 3904
0xBB: 0 3904
0xBC: 0 3904
0xBD: 0 3904
0xBE: 0 3904
0xBF: 0 3904
0xC0: 10872 3911
0xC1: 47712 3906
0xC2: 462 3911
0xC3: 0 3906
0xC4: 0 3906
0xC5: 0 3911
0xC6: 0 3906
0xC7: 0 3911
0xC8: 0 3904
0xC9: 0 3904
0xCA: 0 3904
0xCB: 0 3904
0xCC: 0 3904
0xCD: 0 3904
0xCE: 0 3904
0xCF: 8232 3904
0xD0: 50877 3911
0xD1: 792 3906
0xD2: 0 3911
0xD3: 0 3906
0xD4: 0 3906
0xD5: 0 3911
0xD6: 0 3906
0xD7: 0 3911
0xD8: 0 3904
0xD9: 0 3904
0xDA: 0 3904
0xDB: 0 3904
0xDC: 0 3904
0xDD: 0 3904
0xDE: 6062 3904
0xDF: 53262 3904
0xE0: 1287 3906
0xE1: 0 3911
0xE2: 0 3906
0xE3: 0 3911
0xE4: 0 3911
0xE5: 0 3906
0xE6: 0 3911
0xE7: 0 3906
0xE8: 0 3904
0xE9: 0 3904
0xEA: 0 3904
0xEB: 0 3904
0xEC: 0 3904
0xED: 4332 3904
0xEE: 54747 3904
0xEF: 2002 3904
0xF0: 0 3906
0xF1: 0 3911
0xF2: 0 3906
0xF3: 0 3911
0xF4: 0 3911
0xF5: 0 3906
0xF6: 0 3911
0xF7: 0 3906
0xF8: 0 3904
0xF9: 0 3904
0xFA: 0 3904
0xFB: 0 3904
0xFC: 2997 3904
0xFD: 55252 3904
0xFE: 2997 3904
0xFF: 0 3904

--
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Isaac M. Bavaresco]

Em 12/12/2017 06:35, Brent Brown escreveu:
> For my previously mentioned requirement of the check value not equalling 0xFF, I
> think I can live with testing for and replacing that occurence with a different code.

Did you see my other reply about spreading the CRC or checksum over two
bytes? That way you can ensure you will never have a 0xff.
But by using the CRC you don't have to worry about having a 0xff value
for CRC or even for the data, because CRC will catch virtually any data
corruption.
If you use BCD for your data you will use half the storage space, which
equals to half EEPROM writing time too.


And much better than using just a "dirty" flag, adding a sequence count
to each record allows you to find the latest valid one.
You could use a record with 2-byte sequence counter, 3-byte BCD value
plus 1-byte CRC. The CRC calculated over the sequence counter and data,
so it would be easy to spot corrupted records.

It would be wise to skip the 0xffff sequence value though.

Cheers,
Isaac


---
Este email foi escaneado pelo Avast antivírus.
https://www.avast.com/antivirus


--
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
View/change your membership opti

[Brent Brown]

On 12 Dec 2017 at 9:44, Isaac M. Bavaresco wrote:

> > For my previously mentioned requirement of the check value not equalling 0xFF, I
> > think I can live with testing for and replacing that occurence with a different code.
>
> Did you see my other reply about spreading the CRC or checksum over two
> bytes? That way you can ensure you will never have a 0xff.
> But by using the CRC you don't have to worry about having a 0xff value
> for CRC or even for the data, because CRC will catch virtually any data
> corruption.
> If you use BCD for your data you will use half the storage space, which
> equals to half EEPROM writing time too.

I could spread the CRC8 value over 2 bytes, similar to what I've done with the digits.
Though I don't see any major problem with sticking to 1 byte and translating a CRC
of 0xFF into say 0x5A. It does feel a little cheeky, but it's not like I'm transmitting the
data between devices... the CRC function will be literally the same code on the
same device for both reading and writing.

Yes I could have packed 2 BCD digits into one byte, but have opted to spread the
data wider to give me an extra method of corruption detection. I can test 3 ways
now: check the high/low nibbles to see if a symbol for a single digit is valid, check
the CRC over all 6 digits, and I can also check against the previous record to
ensure it has incremented by only 1 count.

> And much better than using just a "dirty" flag, adding a sequence count
> to each record allows you to find the latest valid one.
> You could use a record with 2-byte sequence counter, 3-byte BCD value
> plus 1-byte CRC. The CRC calculated over the sequence counter and data,
> so it would be easy to spot corrupted records.

I'm not sure that I'll use a dirty flag... if need be I can change data so the checks fail
and that way mark the record as bad/dirty/deleted.

A sequence count would increase just like the hours count does, and at some point
it will roll over as well. So if I have to handle a rollover eventually, then I may as well
just handle that for the hours count alone. The thought is that marking old records
"dirty" on roll-over would be one way of handling it.

Thanks.

--
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Isaac M. Bavaresco]

The mechanism of the sequence count works differently than the time
roll-over.

With two bytes you will have 65536 possible values, but your EEPROM
memory has room for, say, 128 records.

So if any record has sequence number more than 128 counts AHEAD of the
one you have, it is older. We could increase (double it, for instance)
this threshold to account for some lost record in-between.

You compare sequence numbers by subtracting them and doing an unsigned
compare:

if( (uint16_t)( Seq2 - Seq1 ) < THRESHOLD )
    {
    /* Record with sequence Seq2 is newer than record with sequence Seq1 */
    }

This method takes care of roll-over, you will never need to check for
overflow.


Example:

#define THRESHOLD    256

int    NewestRecord    = -1;
for( int i = 0; i < NumRecords; i++ )
    {
    if( ValidateRecord( i ))
        {
        if( NewestRecord < 0 || (uint16_t)( GetSequence( i ) -
GetSequence( NewestRecord )) < THRESHOLD )
            NewestRecord    = i;
        }
    }

if( NewestRecord < 0 )
    {
    /* There is no valid record. */
    }
else
    {
    /* This is the record with the latest saved value. */
    }


Cheers,
Isaac



Em 12/12/2017 11:01, Brent Brown escreveu:
> On 12 Dec 2017 at 9:44, Isaac M. Bavaresco wrote:
>
>>> For my previously mentioned requirement of the check value not equalling 0xFF, I
>>> think I can live with testing for and replacing that occurence with a different code.
>> Did you see my other reply about spreading the CRC or checksum over two
>> bytes? That way you can ensure you will never have a 0xff.
>> But by using the CRC you don't have to worry about having a 0xff value
>> for CRC or even for the data, because CRC will catch virtually any data
>> corruption.
>> If you use BCD for your data you will use half the storage space, which
>> equals to half EEPROM writing time too.
> I could spread the CRC8 value over 2 bytes, similar to what I've done with the digits.
> Though I don't see any major problem with sticking to 1 byte and translating a CRC
> of 0xFF into say 0x5A. It does feel a little cheeky, but it's not like I'm transmitting the
> data between devices... the CRC function will be literally the same code on the
> same device for both reading and writing.
>
> Yes I could have packed 2 BCD digits into one byte, but have opted to spread the
> data wider to give me an extra method of corruption detection. I can test 3 ways
> now: check the high/low nibbles to see if a symbol for a single digit is valid, check
> the CRC over all 6 digits, and I can also check against the previous record to
> ensure it has incremented by only 1 count.
>
>> And much better than using just a "dirty" flag, adding a sequence count
>> to each record allows you to find the latest valid one.
>> You could use a record with 2-byte sequence counter, 3-byte BCD value
>> plus 1-byte CRC. The CRC calculated over the sequence counter and data,
>> so it would be easy to spot corrupted records.
> I'm not sure that I'll use a dirty flag... if need be I can change data so the checks fail
> and that way mark the record as bad/dirty/deleted.
>
> A sequence count would increase just like the hours count does, and at some point
> it will roll over as well. So if I have to handle a rollover eventually, then I may as well
> just handle that for the hours count alone. The thought is that marking old records
> "dirty" on roll-over would be one way of handling it.
>
> Thanks.
>


---
Este email foi escaneado pelo Avast antivírus.
https://www.avast.com/antivirus

--
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/p

[Brent Brown]

On 12 Dec 2017 at 11:31, Isaac M. Bavaresco wrote:

> The mechanism of the sequence count works differently than the time
> roll-over.
>
> With two bytes you will have 65536 possible values, but your EEPROM
> memory has room for, say, 128 records.
>
> So if any record has sequence number more than 128 counts AHEAD of the
> one you have, it is older. We could increase (double it, for instance)
> this threshold to account for some lost record in-between.
>
> You compare sequence numbers by subtracting them and doing an unsigned
> compare:
>
> if( (uint16_t)( Seq2 - Seq1 ) < THRESHOLD )
>     {
>     /* Record with sequence Seq2 is newer than record with sequence Seq1 */
>     }
>
> This method takes care of roll-over, you will never need to check for
> overflow.
>
> Example:
>
> #define THRESHOLD    256
>
> int    NewestRecord    = -1;
> for( int i = 0; i < NumRecords; i++ )
>     {
>     if( ValidateRecord( i ))
>         {
>         if( NewestRecord < 0 || (uint16_t)( GetSequence( i ) -
> GetSequence( NewestRecord )) < THRESHOLD )
>             NewestRecord    = i;
>         }
>     }
>
> if( NewestRecord < 0 )
>     {
>     /* There is no valid record. */
>     }
> else
>     {
>     /* This is the record with the latest saved value. */
>     }
>

Ok, cool, I see how that works, nice.

For that matter though it is conceivable that the record counter and the hours
counter could be the same thing. Kind of what I was thinking all along but probably
never spelled it out. Count increments by a value of one each time interval, store
record in next EEPROM location(s). At power up find the highest record
number/hours count using the method you descibe.

The only slight challenge I will have, and not mentioned previously, is that a service
person has access to change the engine hours count - for example when servicing
a machine and swapping out this module. In that one and only case the "new" hours
count could be lower than any of the "old" records stored in EEPROM, or "ahead"
by more than the number of records stored. The solution would be to delete, or
mark as invalid, all the old values at that time. I don't like to have code that does
this, with the chance that it could run errantly, but concede it may be necessary. I
have safeguards in place to reduce the chances of the service person routines
being accessed improperly.
--
Electronic Design Solutions
9 Titoki Place, Pukete
Hamilton 3200, New Zealand
P: [phone]
M: [phone]



--
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[mbaum7901]

Your comment on the servicing reminded me of something that I normally do on
this type of product: I maintain time like items as a running total from 0
and separately record what time 0 is. In your case, if the during servicing
they failed to transfer the running time to the new unit, that could be
rectified when that was subsequently discovered assuming of course that
there was a paper record of the device swap. The display then become adding
the running time plus the starting offset or base time. And if default
offset is 0xffff.. rather than 0 then failing to initialize the device is
evident.


-----Original Message-----
From: piclist-***@mit.edu [mailto:piclist-***@mit.edu] On Behalf Of
Brent Brown
Sent: Tuesday, December 12, 2017 15:20
To: Microcontroller discussion list - Public. <***@mit.edu>
Subject: Re: [PIC] Sum of digits

On 12 Dec 2017 at 11:31, Isaac M. Bavaresco wrote:

> The mechanism of the sequence count works differently than the time
> roll-over.
>
> With two bytes you will have 65536 possible values, but your EEPROM
> memory has room for, say, 128 records.
>
> So if any record has sequence number more than 128 counts AHEAD of the
> one you have, it is older. We could increase (double it, for instance)
> this threshold to account for some lost record in-between.
>
> You compare sequence numbers by subtracting them and doing an unsigned
> compare:
>
> if( (uint16_t)( Seq2 - Seq1 ) < THRESHOLD )
>     {
>     /* Record with sequence Seq2 is newer than record with sequence
> Seq1 */
>     }
>
> This method takes care of roll-over, you will never need to check for
> overflow.
>
> Example:
>
> #define THRESHOLD    256
>
> int    NewestRecord    = -1;
> for( int i = 0; i < NumRecords; i++ )
>     {
>     if( ValidateRecord( i ))
>         {
>         if( NewestRecord < 0 || (uint16_t)( GetSequence( i ) -
> GetSequence( NewestRecord )) < THRESHOLD )
>             NewestRecord    = i;
>         }
>     }
>
> if( NewestRecord < 0 )
>     {
>     /* There is no valid record. */
>     }
> else
>     {
>     /* This is the record with the latest saved value. */
>     }
>

Ok, cool, I see how that works, nice.

For that matter though it is conceivable that the record counter and the
hours counter could be the same thing. Kind of what I was thinking all along
but probably never spelled it out. Count increments by a value of one each
time interval, store record in next EEPROM location(s). At power up find the
highest record number/hours count using the method you descibe.

The only slight challenge I will have, and not mentioned previously, is that
a service person has access to change the engine hours count - for example
when servicing a machine and swapping out this module. In that one and only
case the "new" hours count could be lower than any of the "old" records
stored in EEPROM, or "ahead"
by more than the number of records stored. The solution would be to delete,
or mark as invalid, all the old values at that time. I don't like to have
code that does this, with the chance that it could run errantly, but concede
it may be necessary. I have safeguards in place to reduce the chances of the
service person routines being accessed improperly.
--
Electronic Design Solutions
9 Titoki Place, Pukete
Hamilton 3200, New Zealand
P: [phone]
M: [phone]



--
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive View/change
your membership options at http://mailman.mit.edu/mailman/listinfo/piclist


--
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Isaac M. Bavaresco]

Em 12/12/2017 11:01, Brent Brown escreveu:
> I could spread the CRC8 value over 2 bytes, similar to what I've done with the digits.
> Though I don't see any major problem with sticking to 1 byte and translating a CRC
> of 0xFF into say 0x5A. It does feel a little cheeky, but it's not like I'm transmitting the
> data between devices... the CRC function will be literally the same code on the
> same device for both reading and writing.


This way, anytime you find a saved CRC value of 0x5A, you will have to
accept the data if the calculated CRC is 0xFF too.
It will weaken the confidence on your data.

Cheers,
Isaac


---
Este email foi escaneado pelo Avast antivírus.
https://www.avast.com/antivirus


--
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/list

[Brent Brown]

On 12 Dec 2017 at 11:37, Isaac M. Bavaresco wrote:

> > I could spread the CRC8 value over 2 bytes, similar to what I've done with the digits.
> > Though I don't see any major problem with sticking to 1 byte and translating a CRC
> > of 0xFF into say 0x5A. It does feel a little cheeky, but it's not like I'm transmitting the
> > data between devices... the CRC function will be literally the same code on the
> > same device for both reading and writing.
>
> This way, anytime you find a saved CRC value of 0x5A, you will have to
> accept the data if the calculated CRC is 0xFF too.
> It will weaken the confidence on your data.

Agreed, there were 1 in 256 patterns in 10^6 or 0.36% that have the CRC value
0x5A, now there will be twice as many: 2 in 256 or 0.78%. It doesn't change the
odds for the other 254 out of 256 possible CRC values, so not a huge weakness.

Just reminding myself why I am doing this: By avoiding 0xFF as a CRC value I
avoid incorrectly reading as valid a record where the data was written to EEPROM
but the CRC was never written (which normally happens after validation), with 0xFF
simply being the erased state of the EEPROM.

--
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Isaac M. Bavaresco]

Em 12/12/2017 19:17, Brent Brown escreveu:
> Just reminding myself why I am doing this: By avoiding 0xFF as a CRC value I
> avoid incorrectly reading as valid a record where the data was written to EEPROM
> but the CRC was never written (which normally happens after validation), with 0xFF
> simply being the erased state of the EEPROM.

You are missing one important point: For a record be deemed valid, the
CRC calculated during the checking procedure must be equal to the CRC
that was calculated during the saving process (and that was saved along
with the data).

It is impossible for a record that does not have its CRC value stored
with it to pass the validity test, unless the random value in the CRC
area is equal to the correct CRC (but then, it doesn't matter if it was
not saved,it was already there).

Cheers,
Isaac


---
Este email foi escaneado pelo Avast antivírus.
https://www.avast.com/antivirus


--
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
View/change your membership options

[Brent Brown]

On 12 Dec 2017 at 23:30, Isaac M. Bavaresco wrote:

> > Just reminding myself why I am doing this: By avoiding 0xFF as a CRC value I
> > avoid incorrectly reading as valid a record where the data was written to EEPROM
> > but the CRC was never written (which normally happens after validation), with 0xFF
> > simply being the erased state of the EEPROM.
>
> You are missing one important point: For a record be deemed valid, the
> CRC calculated during the checking procedure must be equal to the CRC
> that was calculated during the saving process (and that was saved along
> with the data).
>
> It is impossible for a record that does not have its CRC value stored
> with it to pass the validity test, unless the random value in the CRC
> area is equal to the correct CRC (but then, it doesn't matter if it was
> not saved,it was already there).

Hi Isaac,

It is that random value in the CRC position that concerns me. If it was not
intentionally written there then the record was never "validated", yet it could
accidentally match the correct CRC and make it appear valid. A little more likely to
occur with a CRC of 0xFF, due to that being the erased state of EEPROM, but then
probably only until one full cycle through EEPROM has altered all the values.

I imagine the process will go like this: I save a few bytes of data to EEPROM, I
immediately read it back to verify that it has been written correctly, if verified then I
write the CRC to the immediately following address in EEPROM, reading back
again to verify the whole record. If the CRC value is not written (could be because
the data was corrupt, or failed to write correctly, and failed the first verification), and
the EEPROM location where the CRC should have been written accidentally
matches the correct CRC, then it becomes a "valid" record which I don't want it to
be.

If I exclude 0xFF from the list of valid CRC's as planned, and instead use that value
to indicate "no CRC, record not validated", then record validation can be better
trusted. It may be advantageous to write 0xFF to the CRC location one record
ahead of the record being written... though that would double the number of writes
to those locations and impact on EEPROM endurance.

Cheers, Brent.

--
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Isaac M. Bavaresco]

I think you are overthinking it.

The usual way of doing things is to calculate the CRC before writing the
data and writing the entire record at once, data and CRC. As you are
using EEPROM, it is possible to write the CRC first, so if a write is
aborted midways then there is no possibility of having data without a
CRC in the memory. Or simply put the CRC in the beginning of the record.

Think of a block memory device (FLASH, diskette of hard disk drives),
each block or sector must be written in a single operation.

If you do things this way then your concern becomes a lot less important.

It is always possible for a memory corruption to alter data and CRC at
the same time and make a corrupted record to appear OK, but it is highly
improbable.

One measure that I think is easy to implement and adds a lot of security
is avoiding sequence number 0xffff.

Cheers,
Isaac





Em 13/12/2017 01:06, Brent Brown escreveu:
> On 12 Dec 2017 at 23:30, Isaac M. Bavaresco wrote:
>
>>> Just reminding myself why I am doing this: By avoiding 0xFF as a CRC value I
>>> avoid incorrectly reading as valid a record where the data was written to EEPROM
>>> but the CRC was never written (which normally happens after validation), with 0xFF
>>> simply being the erased state of the EEPROM.
>> You are missing one important point: For a record be deemed valid, the
>> CRC calculated during the checking procedure must be equal to the CRC
>> that was calculated during the saving process (and that was saved along
>> with the data).
>>
>> It is impossible for a record that does not have its CRC value stored
>> with it to pass the validity test, unless the random value in the CRC
>> area is equal to the correct CRC (but then, it doesn't matter if it was
>> not saved,it was already there).
> Hi Isaac,
>
> It is that random value in the CRC position that concerns me. If it was not
> intentionally written there then the record was never "validated", yet it could
> accidentally match the correct CRC and make it appear valid. A little more likely to
> occur with a CRC of 0xFF, due to that being the erased state of EEPROM, but then
> probably only until one full cycle through EEPROM has altered all the values.
>
> I imagine the process will go like this: I save a few bytes of data to EEPROM, I
> immediately read it back to verify that it has been written correctly, if verified then I
> write the CRC to the immediately following address in EEPROM, reading back
> again to verify the whole record. If the CRC value is not written (could be because
> the data was corrupt, or failed to write correctly, and failed the first verification), and
> the EEPROM location where the CRC should have been written accidentally
> matches the correct CRC, then it becomes a "valid" record which I don't want it to
> be.
>
> If I exclude 0xFF from the list of valid CRC's as planned, and instead use that value
> to indicate "no CRC, record not validated", then record validation can be better
> trusted. It may be advantageous to write 0xFF to the CRC location one record
> ahead of the record being written... though that would double the number of writes
> to those locations and impact on EEPROM endurance.
>
> Cheers, Brent.
>


---
Este email foi escaneado pelo Avast antivírus.
https://www.avast.com/antivirus


--
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclis

[Brent Brown]

On 13 Dec 2017 at 12:13, Isaac M. Bavaresco wrote:

> I think you are overthinking it.
>
> The usual way of doing things is to calculate the CRC before writing the
> data and writing the entire record at once, data and CRC. As you are
> using EEPROM, it is possible to write the CRC first, so if a write is
> aborted midways then there is no possibility of having data without a
> CRC in the memory. Or simply put the CRC in the beginning of the record.
>
> Think of a block memory device (FLASH, diskette of hard disk drives),
> each block or sector must be written in a single operation.
>
> If you do things this way then your concern becomes a lot less important.
>
> It is always possible for a memory corruption to alter data and CRC at
> the same time and make a corrupted record to appear OK, but it is highly
> improbable.
>
> One measure that I think is easy to implement and adds a lot of security
> is avoiding sequence number 0xffff.

I think you are right, I have been over thinking it. Will simplify my approach,
implement it, test it, and let you know the result. Many thanks for your input.

--
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Nicola Perotto]

Hi Brent,

On 11/12/2017 01:52, Brent Brown wrote:
> ...
>
> This results in 10 symbols for numbers 0-9
> as follows: 0x3C, 0x4B, 0x5A, 0x69, 0x78, 0x87, 0x96, 0xA5, 0xB4, 0xC3.
> ...
> Preferably
> I'd like the checksum to never equal FF, as that is the "erased" value of EEPROM, I
> will also use that as a "start of record" marker. Thus a complete record will be 8
> bytes:
>
> start_record, digit1, digit2, digit3, digit4, digit5, digit6, checksum
>
> Example: (0000.01 hours)
>
> 0xFF, 0x3C, 0x3C, 0x3C, 0x3C, 0x3C, 0x4B, 0x77
>
> ...
I would use a start_record value different from the "erased" value.
And I would use two so if one I get a checksum of 0xFF, I can swap the
start_record and accomodate the checksum.
     Nicola

--
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
View/change your membership options at
http://

[p***@roadrunner.com]

You could just throw away one bit of checksum by forcing the MSBit to zero.

- - Bob Ammerman
RAm Systems

-----Original Message-----
From: piclist-***@mit.edu [mailto:piclist-***@mit.edu] On Behalf Of
Brent Brown
Sent: Sunday, December 10, 2017 7:52 PM
To: ***@mit.edu
Subject: [PIC] Sum of digits

Hi all,

I have a math problem that is a little interesting. Background first: I'm
storing "operating hours" in internal EEPROM on a PIC16F887. It's in a
product I've been making for some time, but despite my best efforts so far
the hours counter occasionally gets corrupted. I suspect the biggest problem
is writing to EEPROM in a "dying gasp" when power supply falls, with simply
not enough time to reliably do so.

A new scheme is planned. Hours increment in units 0.01hr (36s) up to
9999.99, then roll over to 0. On each increment it will be stored in a
different EEPROM location. This addresses three issues 1) the byte endurance
of EEPROM (1M typ but 100k guaranteed), 2) ability to retrieve a previous
record if corruption is detected, loosing as little as 36 seconds (fairly
insignificant), and 3) no longer writing to EEPROM as power fails.

I plan to encode the number essentially as 6 x BCD digits, 1 digit per byte.
I take the BCD value, add 3 to it (avoids 0 being a valid code), shift to
upper nibble, and write the inverted value into the lower nibble. This
results in 10 symbols for numbers 0-9 as follows: 0x3C, 0x4B, 0x5A, 0x69,
0x78, 0x87, 0x96, 0xA5, 0xB4, 0xC3.

Corruption in a digit is detected when a byte does not equate to one of the
valid symbols (10 valid 8 bit codes out of 256). In addition I'll add a
checksum. Preferably I'd like the checksum to never equal FF, as that is the
"erased" value of EEPROM, I will also use that as a "start of record"
marker. Thus a complete record will be 8
bytes:

start_record, digit1, digit2, digit3, digit4, digit5, digit6, checksum

Example: (0000.01 hours)

0xFF, 0x3C, 0x3C, 0x3C, 0x3C, 0x3C, 0x4B, 0x77

My curious math question is can the checksum ever be 0xFF? It's complicated
by the values I'm using for the symbols, and the checksum being the LS 8
bits of the sum.

Simplifying the symbols back to decimal values the question becomes: in the
sequence 000000 - 999999 what are all the possible values for the sum of
digits?
One can see fairly easily that the minimum would be 0+0+0+0+0+0=0, the
maximum would be 9+9+9+9+9+9=54. Also: there are 1 million possible sums
(10^6), all will be in the range 0-54 (9*6), all integer values will be
covered as the step size is 1, and there will be many identical results.

Wondering if/how I could solve this mathematically. I did make a spreadsheet
with
1000 rows to calculate the sums of digits from 000-999 and graphed the
results in a histogram. 1,000,000 rows is possible in Excel, but takes a bit
of time, working on that now. Interestingly, the 1000 row spreadsheet showed
sum of digits covered all values from 0-27, as expected, and the
distribution of values is a bell curve.

I can of course work around and allow for the checksum being 0xFF, or change
the value if/when it ever does match 0xFF, but still curious to know if it
is necessary.


--
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

--
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist

[Isaac M. Bavaresco]

Spread your checksum over two bytes, say: checksum is 0Xmn -> convert to
0XmA:0X5n, for instance. Or better, use 0X(m:~m):0X(n:~n).

For the corruption problem, you could use an early power-fail detection
circuit that triggers an interrupt that writes the data only at power-down.
That has the additional advantage of only writing the EEPROM once each
power-down. I used this method already and if implemented correctly is
very good.

Better than writing always to the same addresses, use an approach like
Microchip's EEPROM emulation application note, write a sequence number
together with each record, so you can search the memory at each power-up
to find which is the newest value and start from it.
This method has the extra benefit of spreading the wear over the entire
EEPROM.

Cheers,
Isaac



Em 10/12/2017 22:52, Brent Brown escreveu:
> Hi all,
>
> I have a math problem that is a little interesting. Background first: I'm storing
> "operating hours" in internal EEPROM on a PIC16F887. It's in a product I've been
> making for some time, but despite my best efforts so far the hours counter
> occasionally gets corrupted. I suspect the biggest problem is writing to EEPROM in
> a "dying gasp" when power supply falls, with simply not enough time to reliably do
> so.
>
> A new scheme is planned. Hours increment in units 0.01hr (36s) up to 9999.99,
> then roll over to 0. On each increment it will be stored in a different EEPROM
> location. This addresses three issues 1) the byte endurance of EEPROM (1M typ
> but 100k guaranteed), 2) ability to retrieve a previous record if corruption is
> detected, loosing as little as 36 seconds (fairly insignificant), and 3) no longer
> writing to EEPROM as power fails.
>
> I plan to encode the number essentially as 6 x BCD digits, 1 digit per byte. I take the
> BCD value, add 3 to it (avoids 0 being a valid code), shift to upper nibble, and write
> the inverted value into the lower nibble. This results in 10 symbols for numbers 0-9
> as follows: 0x3C, 0x4B, 0x5A, 0x69, 0x78, 0x87, 0x96, 0xA5, 0xB4, 0xC3.
>
> Corruption in a digit is detected when a byte does not equate to one of the valid
> symbols (10 valid 8 bit codes out of 256). In addition I'll add a checksum. Preferably
> I'd like the checksum to never equal FF, as that is the "erased" value of EEPROM, I
> will also use that as a "start of record" marker. Thus a complete record will be 8
> bytes:
>
> start_record, digit1, digit2, digit3, digit4, digit5, digit6, checksum
>
> Example: (0000.01 hours)
>
> 0xFF, 0x3C, 0x3C, 0x3C, 0x3C, 0x3C, 0x4B, 0x77
>
> My curious math question is can the checksum ever be 0xFF? It's complicated by
> the values I'm using for the symbols, and the checksum being the LS 8 bits of the
> sum.
>
> Simplifying the symbols back to decimal values the question becomes: in the
> sequence 000000 - 999999 what are all the possible values for the sum of digits?
> One can see fairly easily that the minimum would be 0+0+0+0+0+0=0, the
> maximum would be 9+9+9+9+9+9=54. Also: there are 1 million possible sums
> (10^6), all will be in the range 0-54 (9*6), all integer values will be covered as the
> step size is 1, and there will be many identical results.
>
> Wondering if/how I could solve this mathematically. I did make a spreadsheet with
> 1000 rows to calculate the sums of digits from 000-999 and graphed the results in a
> histogram. 1,000,000 rows is possible in Excel, but takes a bit of time, working on
> that now. Interestingly, the 1000 row spreadsheet showed sum of digits covered all
> values from 0-27, as expected, and the distribution of values is a bell curve.
>
> I can of course work around and allow for the checksum being 0xFF, or change the
> value if/when it ever does match 0xFF, but still curious to know if it is necessary.
>
>


---
Este email foi escaneado pelo Avast antivírus.
https://www.avast.com/antivirus


--
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
View/change your membership options at
http://mailman