"new release" message contains a non-ASCII character

[RobMeades]

Bug Report

If arduino-cli detects that a new version is available it says so, which is great, but the message includes a non-ASCII character (a little right-arrow) and so, where arduino-cli is called from a Python [3.8.5] script, it will barf on it, which is non-optimal, i.e. all our automation suddenly stops working when a new version of arduino-cli is available!

It would be better to remove this character.

FYI, I've tried adding --no-colour but that only removes the escape sequences from either side, not the non-ASCII character.

Current behavior

2021-09-29_09:34:14: arduino-cli, pid 26304 started with guard time 300 second(s)
2021-09-29_09:34:15: arduino-cli alpha Version: 0.19.0 Commit: 56419ecd Date: 2021-09-02T14:47:35Z
2021-09-29_09:34:15: A new release of Arduino CLI is available: 0.19.0 → 0.19.1
2021-09-29_09:34:15: failed: UnicodeEncodeError while trying to execute 'charmap' codec can't encode character '\u2192' in position 88: character maps to <undefined>.

You can see the non-ASCII character towards the right-hand end of the third line.

Expected behavior

Be ASCII please.

Environment

• CLI version: see above
• OS and platform: Win10 pro 20H2

[silvanocerza]

Am not sure we want to limit the Arduino CLI to use only ASCII, UTF has been around for some time.

In any case if you don't want to see the new release notification you can disable it by setting the updater.disable_notification config or the env var ARDUINO_UPDATER_DISABLE_NOTIFICATION to true.

[RobMeades]

Up to you of course, but an essentially unnecessary non-ASCII character appearing in the middle of a string from a command-line application is quite likely to cause people unexpected issues.

[silvanocerza]

I wouldn't expect any modern terminal or language to have issues with UTF though, neither I would expect people try to parse the text format output of the CLI, that's meant for humans.

If you want to parse the output of the CLI I strongly suggest using --format json.

[RobMeades]

Not parsing it, just calling it via Python subprocess in order to run it from a Python script and capture the output, just to a log file so that I can see what's going on. Anyway, as I say, up to you, please close this if you don't think it's a problem.

[RobMeades]

Apologies for being a but clueless here but I've tried setting that configuration value as follows:

arduino-cli config add updater.disable_notification true

...but I get back "Settings key doesn't exist". I must have the syntax wrong: can you help me with that please?

[silvanocerza]

My bad! It's updater.enable_notification and ARDUINO_UPDATER_ENABLE_NOTIFICATION, so to set it from command line is:

arduino-cli config set updater.enable_notification false

Not parsing it, just calling it via Python subprocess in order to run it from a Python script and capture the output. Anyway, as I say, up to you, please close this if you don't think it's a problem.

I wouldn't expect it to fail anyway, all our integration tests are written in Python and call the Arduino CLI directly like that. 🤔

[RobMeades]

Thanks! I'll close this now as it's obviously not a problem for others.

[RobMeades]

Just to be clear, in the above I didn't mean "don't do UTF", I meant "don't send non-ASCII codeable characters inside UTF if you don't need to", IYSWIM. Anyway, happy now with the notification disabled.

[RobMeades]

For anyone else who hits this issue, it is to do with Windows 10, or maybe any OS, not having a code-page entry for this particular UTF-8 character in its default code-page table. Obviously the machines that the arduino-cli developers run their Python scripts on don't have this issue but ours (Win10 pro 20H2 in British English) definitely does. The only workaround we found that actually did the trick was to put the output from the Python subprocess that is running arduino-cli through this:

output = process.stdout.readline().decode().encode("ascii", errors="replace").decode()

The reason for the double decode is that we wanted to get the whole line but with the undecodable character replaced with a "?" however, contrary to most of the posts I have seen on the internet, errors="replace" as a parameter to decode() does nothing whatsoever, you have to decode, encode as something nice and simple (we chose ASCII as being sufficient for our purposes since we're only displaying the output for diagnostics) with the undecodable character replaced and then decode() again.