* Trim trailing whitespace in build workflow
* Give build workflow step access to required deployment environment
Certain operations in the "Arduino IDE" GitHub Actions workflow use GitHub Actions "secrets" which are defined in the
repository's administrative settings.
These secrets will typically not be defined when the workflow is run in a fork. However, the workflow's base
functionality, the automated building of the application, does not require secrets. Since that base functionality alone
is very useful to contributors (either to validate relevant changes to the application and infrastructure, or to
generate tester builds) who are performing development work in a fork. For this reason, the workflow is configured to
only perform the secret-dependent operations when the required secrets have been defined in the repository settings.
One such operation is publishing the generated builds to Amazon S3, which Arduino uses to host files for distribution.
This operation depends on the "AWS_ROLE_ARN" secret. As a security measure, this secret is defined inside a deployment
environment (named "production"). GitHub Actions workflow jobs can only use secrets from deployment environments which
they have been explicitly configured to have access to.
At the time the workflow was originally developed, GitHub did not have the deployment environment feature, and so the
workflow was not configured to use environments. The switch to using a deployment environment for this secret was made
only recently, and when that was done, the workflow job that checks whether the secret is defined was not configured to
have access to the "production" environment. This caused the workflow to think it was running in a context where that
secret is not defined even when the secret is in fact defined. The bug caused the workflow to always spuriously skip the
"publish" job which publishes nightly builds of Arduino IDE, and the "publish release" step which publishes production
releases.
The bug is fixed by configuring the "build-type-determination" job so that it has access to the "production"
environment.
* fix(plugin): decouple state update from the LS
To enhance the reliability of Arduino IDE extensions, the update
process for `ArduinoState` has been modified to ensure independence
from the language server's availability. This change addresses issues
caused by `compileSummary` being `undefined` due to potential startup
failures of the Arduino Language Server, as noted in
https://github.com/dankeboy36/esp-exception-decoder/issues/28#issuecomment-2681800772.
The `compile` command now resolves with a `CompileSummary` rather than
`void`, facilitating a more reliable way for extensions to access
necessary data. Furthermore, the command has been adjusted to allow
resolution with `undefined` when the compiled data is partial.
By transitioning to direct usage of the resolved compile value for
state updates, the reliance on executed commands for extensions is
eliminated. This update also moves the VSIX command execution to the
frontend without altering existing IDE behavior.
Closesarduino/arduino-ide#2642
Signed-off-by: dankeboy36 <dankeboy36@gmail.com>
* fix: install missing libx11-dev and libxkbfile-dev
Signed-off-by: dankeboy36 <dankeboy36@gmail.com>
* fix: pick better GH step name
Signed-off-by: dankeboy36 <dankeboy36@gmail.com>
* fix: install the required dependencies on Linux
Signed-off-by: dankeboy36 <dankeboy36@gmail.com>
* fix(revert): do not manually install deps on Linux
Signed-off-by: dankeboy36 <dankeboy36@gmail.com>
* chore: pin `ubuntu-22.04` for linux actions
* fix: restore accidentally removed dispose on finally
Signed-off-by: dankeboy36 <dankeboy36@gmail.com>
* fix(test): align mock naming 💄
Signed-off-by: dankeboy36 <dankeboy36@gmail.com>
* fix: let the ino contribution notify the LS
+ event emitter dispatches the new state.
Signed-off-by: dankeboy36 <dankeboy36@gmail.com>
* fix(test): emit the new compiler summary state
Signed-off-by: dankeboy36 <dankeboy36@gmail.com>
* chore(revert): unpin linux version, use latest
revert of b11bde1c47
Signed-off-by: dankeboy36 <dankeboy36@gmail.com>
---------
Signed-off-by: dankeboy36 <dankeboy36@gmail.com>
Co-authored-by: Giacomo Cusinato <7659518+giacomocusinato@users.noreply.github.com>
The Arduino IDE installation includes a collection of example sketches demonstrating fundamental concepts.
These examples are hosted in a dedicated repository, which is a dependency of this project. A new release has been made
in that `arduino/arduino-examples` repository.
The infrastructure for downloading the examples during the Arduino IDE build is hereby updated to use the latest release
of the `arduino/arduino-examples` repository.
Install `libx11-dev`, `libxkbfile-dev`, `libsecret-1-dev` libraries as the most recent update to ubuntu-latest does not include them
Signed-off-by: dankeboy36 <dankeboy36@gmail.com>
Background
==========
Shared Library Dependencies
---------------------------
The Linux build of Arduino IDE has dynamic linkage against the libstdc++ and glibc shared libraries. This results in
it having a dependency on the version of the libraries that happens to be present in the environment it is built in.
Although newer versions of the shared libraries are compatible with executables linked against an older version, the
reverse is not true. This means that building Arduino IDE on a Linux machine with a recent distro version installed
causes the IDE to error on startup for users who have a distro with older versions of the dependencies.
For example, if Arduino IDE were built on a machine with version 3.4.33 of libstdc++, then attempting to run it on a
machine with an older version of libstdc++ would fail with an error like:
```
Error: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.33' not found (required by /home/foo/arduino-ide/resources/app/lib/backend/native/nsfw.node)
```
Likewise, if Arduino IDE were built on a machine with version 2.39 of glibc, then attempting to run it on a machine
with an older version of glibc would fail with an error like:
```
Error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.39' not found (required by /home/foo/arduino-ide/resources/app/node_modules/nsfw/build/Release/nsfw.node)
```
Build Machine Requirements
--------------------------
The IDE builds distributed by Arduino should be compatible with a reasonable range of Linux distribution versions. In
order to achieve this, the builds must be performed in a machine with an older version of the shared libraries. The
shared libraries are part of the Linux distro, and installing a different version is not feasible. So this imposes a
maximum limit on the build machine's distro version.
The distributed builds are generated via a GitHub Actions workflow. The most simple approach is to run the build in the
machine of the GitHub-hosted runners provided for each operating system. However, GitHub provides a limited range of
operating system versions in their runners, and removes the older versions as newer versions are added. This means that
building in the GitHub-hosted runner machine would not allow for the desired range of Linux distro version
compatibility. For this reason, the Linux build is performed in a Docker container that provides an older version of
Ubuntu.
The same situation of incompatibility with Linux distro versions that have a version of the shared library dependencies
older than the version present on the build machine occurs for several of the tools and frameworks used by the build
process (e.g., Node.js, Python). In this case, the tables are turned as we are now the user rather than the distributor
and so are at the mercy of the Linux distro version compatibility range provided by the distributor. So this imposes a
minimum limit on the build machine's distro version.
Although several of the dependencies used by the standard build system have dependencies on versions of glibc higher
than the version 2.27 present in Ubuntu 18.04, it was possible to use this distro version in the Linux build container
by using alternative distributions and/or versions of these dependencies.
Workflow Artifacts
------------------
The build workflow uses GitHub actions workflow artifacts to transfer the files generated by the build job to subsequent
jobs in the workflow. The "actions/upload-artifact" action is used for this purpose.
Problem
=======
GitHub is dropping support for the workflow artifacts produced by the version 3.x of the "actions/upload-artifact"
action that was previously used by the build job. So the action version used in the build workflow was updated to the
current version 4.x. This version of the action uses a newer version of the Node.js runtime (20). Unfortunately the the
Node.js 20 runtime used by the action has a dependency on glibc version 2.28, which causes the Linux build job to fail
after the update of the "actions/upload-artifact" action:
```
Run actions/upload-artifact@v4
/__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node)
```
Unlike the other dependencies of the build process, it is no longer possible to work around this incompatibility by
continuing to use the older compatible version of the "actions/upload-artifact" action. It is also impossible to replace
the incompatible Node.js 20.x distribution used by the action, since it comes from the read-only file system of the
runner image. Likewise, it is not possible to configure or force the action to use a Node.js installation at a different
path on the runner machine.
Resolution
==========
Compatibility with the new version of the "actions/upload-artifact" action is attained by updating the version of Linux
in the build container to 18.10, which is the oldest version that has glibc 2.28. The presence of a newer glibc version
in the container also makes it compatible with several other dependencies of the build process, meaning the code in the
Dockerfile and workflow for working around the incompatibilities of Ubuntu 18.04 can be removed.
Consequences
============
Unfortunately this means the loss of compatibility of the Linux Arduino IDE builds with distros that use glibc 2.27
(e.g., Ubuntu 18.04). User of those distros will now find that Arduino IDE fails to start with an error like:
```
Error: node-loader:
Error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /home/foo/arduino-ide/resources/app/lib/backend/native/pty.node)
at 85467 (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:2766)
at __webpack_require__ (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:6663105)
at 23571 (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:3374073)
at __webpack_require__ (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:6663105)
at 55444 (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:3369761)
at __webpack_require__ (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:6663105)
at 24290 (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:1780542)
at __webpack_require__ (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:6663105)
at 43416 (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:1770138)
at __webpack_require__ (/home/foo/arduino-ide/resources/app/lib/backend/main.js:2:6663105)
```
The build workflow produces binaries for a range of target hosts. This is done by using a job matrix in the GitHub
Actions workflow that produces each build in a parallel job. GitHub Actions workflow artifacts are used to transfer the
generated files between sequential jobs in the workflow. The "actions/upload-artifact" action is used for this purpose.
Previously, a single artifact was used for this purpose, with each of the parallel jobs uploading its own generated
files to that artifact. However, support for uploading multiple times to a single artifact was dropped in version 4.0.0
of the "actions/upload-artifact" action. So it is now necessary to use a dedicated artifact for each of the builds.
These can be downloaded in aggregate by using the artifact name globbing and merging features which were introduced in
version 4.1.0 of the "actions/download-artifact" action.
The "Sync Labels" GitHub Actions workflow is configured to allow the use of multiple shared label configuration files.
This is done by using a job matrix in the GitHub Actions workflow to download each of the files from the source
repository in a parallel GitHub Actions workflow job. A GitHub Actions workflow artifact was used to transfer the
generated files between sequential jobs in the workflow. The "actions/upload-artifact" and "actions/download-artifact"
actions are used for this purpose.
Previously, a single artifact was used for the transfer of all the shared label configuration files, with each of the
parallel jobs uploading its own generated files to that artifact. However, support for uploading multiple times to a
single artifact was dropped in version 4.0.0 of the "actions/upload-artifact" action. So it is now necessary to use a
dedicated artifact for each of the builds. These can be downloaded in aggregate by using the artifact name globbing and
merging features which were introduced in version 4.1.0 of the "actions/download-artifact" action.
- Use the CLI release proto.zip to get proto files for production versions of CLI
- Extract the proto files from repo if CLI version is declared as `commitsh` or version is 1.1.0
See https://github.com/arduino/arduino-cli/pull/2761
Adds a new preference to control whether the
verify command should automatically run before the
upload. If the `arduino.upload.autoVerify` setting
value is `false`, IDE does not recompile the
sketch code before uploading it to the board.
Signed-off-by: dankeboy36 <dankeboy36@gmail.com>
* fix: use `Status` enum for status code in `ServiceError` type guards
This change resolves the issue where the intersection of `ServiceError` error codes of type `number` resulted in the `never` type due to conflict between number and `State` enum if `StatusObject`
* feat: add `isInvalidArgument` type guard to `ServiceError`
* fix: retry compilation if grpc client needs to be reinitialized
See https://github.com/arduino/arduino-ide/issues/2547
It is considered good practice to use JavaScript's type-safe strict equality operator === instead of the equality
operator ==. Compliance with this practice is enforced by the project's ESLint configuration, via the "eqeqeq" rule.
The script used to generate the changelog for Arduino IDE's auto-update dialog contained an inappropriate usage of the
equality operator. This caused linting runs to fail:
arduino-ide-extension/scripts/compose-changelog.js
37:19 error Expected '===' and instead saw '==' eqeqeq
The `lint` script of the "arduino-ide-extension" package is intended to use the ESLint linter tool to check for problems
in all the package's JavaScript and TypeScript code files. It is used by the continuous integration system to validate
contributions.
Previously, the command invoked `eslint` without any arguments. With the 8.x version of ESLint used by the project, it
is necessary to provide a path argument in order to cause it to lint the contents of files. Because that argument was
not provided, the script didn't do anything at all and so would return a 0 exit status even if the code had linting rule
violations.
This is fixed by adding a `.` path argument to the command invoked by the script. This will cause ESLint to recurse
through the `arduino-ide-extension` folder and lint the code in all relevant files.
The "build" workflow builds the application for all supported targets, generates workflow artifacts from which the
builds can be downloaded by users and beta testers, and publishes nightly and production releases.
As if that wasn't enough, the workflow was also configured to check the sync of the Yarn lockfile.
This monolithic approach is harmful for multiple reasons:
* Makes it difficult to interpret a failed workflow run
* Makes the build workflow more difficult to maintain
* Increases the turnaround time for contributors and maintainers to get feedback from the CI system
The sync check operation is hereby moved to a dedicated workflow, consistent with standard practices for Arduino Tooling
projects.
The "build" workflow builds the application for all supported targets, generates workflow artifacts from which the
builds can be downloaded by users and beta testers, and publishes nightly and production releases.
As if that wasn't enough, the workflow was also configured to perform the unrelated operation of running the project's
test suites.
This monolithic approach is harmful for multiple reasons:
* Makes it difficult to interpret a failed workflow run
* Unnecessarily adds a significant amount of extra content to the already extensive logs produced by the build process
* Makes the build workflow more difficult to maintain
* Increases the length of a build workflow run
* Increases the impact of a spurious failure
* Increases the turnaround time for contributors and maintainers to get feedback from the CI system
The test run operation is hereby moved to a dedicated workflow, consistent with standard practices for Arduino Tooling
projects.
The "build" workflow builds the application for all supported targets, generates workflow artifacts from which the
builds can be downloaded by users and beta testers, and publishes nightly and production releases.
As if that wasn't enough, the workflow was also configured to perform the unrelated operation of linting the project's
TypeScript and JavaScript code.
This monolithic approach is harmful for multiple reasons:
* Makes it difficult to interpret a failed workflow run
* Unnecessarily adds a significant amount of extra content to the already extensive logs produced by the build process
* Makes the build workflow more difficult to maintain
* Increases the length of a build workflow run
* Increases the impact of a spurious failure
* Increases the turnaround time for contributors and maintainers to get feedback from the CI system
The linting operation is hereby moved to a dedicated workflow, consistent with standard practices for Arduino Tooling
projects.
The "build" workflow signs the macOS and Windows builds of the application. The signing process relies on access to GitHub Actions
secrets. For this reason, the workflow is configured to only sign the builds when it has access to GitHub Actions
secrets to avoid spurious failures of the workflow that would otherwise be caused by signing failure.
A flexible general purpose system for determining whether to attempt signing of a build was established years ago. However, a redundant system was added specific to the Windows build instead of using the existing system.
The redundant system is hereby removed. This makes the workflow easier to understand and maintain.
The "build" workflow signs the Windows builds of the application. The signing process relies on access to GitHub Actions
secrets. For this reason, the workflow is configured to only sign the builds when it has access to GitHub Actions
secrets to avoid spurious failures of the workflow that would otherwise be caused by signing failure.
Previously the signing was determined based on the value of the `github.event.pull_request.head.repo.fork` context item.
That was effective for the use case of the workflow being triggered by a pull request from a fork (for security reasons,
GitHub Actions does not give access to secrets under these conditions).
However, there is another context under which the workflow might run without access to the signing secrets, for which
the use of context item is not appropriate. It is important to support the use of the workflow in forks of the
repository. In addition to the possible value to hard forked projects, this is essential to allow conscientious
contributors to test contributions to the build and release system in their own fork prior to submitting a pull request.
The previous configuration would cause a workflow run performed by a contributor in a fork to attempt to sign the
Windows build. Unless the contributor had set up the ridiculously complex infrastructure required to perform the signing
for the Windows build, which is utterly infeasible, this would cause the workflow to fail spuriously.
The appropriate approach, which has been the established convention in the rest of the workflow code, is to use the
secret itself when determining whether to attempt the signing process. If the secret is not defined (resulting in it
having an empty string value), then the signing should be skipped. If it is defined, then the signing should be
performed.
The "build" workflow builds the application for a range of target hosts. This is done by using a job matrix. A separate
parallel job runs for each target. The target-specific configuration data is defined in the job matrix array.
This configuration data includes the information related to the code signing certificates. Inexplicably, during the work
to add support for signing the Windows builds with an "eToken" hardware authentication device, this data was not used
for the Windows code signing configuration. Instead the certificate data was redundantly hardcoded into the workflow
code.
The Windows code signing certificate configuration is hereby changed to use the established flexible job configuration
data system. This makes the workflow easier to understand and maintain.
The Windows builds of the application are cryptographically signed. The signing requires an "eToken" hardware
authentication device be connected to the machine performing the signing. This means that it is necessary to use a
self-hosted GitHub Actions runner for the Windows job of the build workflow rather than the runners hosted by GitHub.
There are some unique characteristics of the self-hosted runner which the workflow code must accommodate. One of these
is that, rather than installing dependencies of the build process during the workflow run as is done for the
GitHub-hosted runners, the dependencies are preinstalled in the self-hosted runner machine. So the dependency
installation steps must be configured so that they will be skipped when the job is running on the self-hosted runner.
This is done by adding a conditional to the steps. Previously the conditional was based on the value of the `runner.os`
context item. This is not an appropriate indicator of the job running on the self-hosted runner because `runner.os` will
have the same value if the job was running on a GitHub-hosted Windows runner. That might seem like only a hypothetical
problem since the workflow does not use a GitHub-hosted Windows runner. However, it is important to support the use of
the workflow in forks of the repository. In addition to the possible value to hard forked projects, this is essential to
allow conscientious contributors to test contributions to the build and release system in their own fork prior to
submitting a pull request.
The conditionals are changed to use the more appropriate indicator of the specific name of the self-hosted Windows
runner (via the `runner.name` context item).
The Windows builds of the application are cryptographically signed. The signing requires an "eToken" hardware
authentication device be connected to the machine performing the signing. This means that it is necessary to use a
self-hosted GitHub Actions runner for the Windows job of the build workflow rather than the runners hosted by GitHub.
There are some unique characteristics of the self-hosted runner which the workflow code must accommodate. The default
working directory of the self-hosted runner is not suitable to perform the build under because the the resulting folder
structure produced paths that exceeded the ridiculously small maximum path length of Windows. So the workflow must be
configured to use a custom working directory with a short path (`C:\a`).
This custom working directory must be used only for the job running on the self-hosted Windows runner so the working
directory of the relevant workflow steps are configured using a ternary expression. Previously, this expression had
multiple conditions:
* the value of the `runner.os` context item
* the definition of a custom working directory value in the job matrix
The second condition is entirely sufficient. The use of the first condition only added unnecessary complexity to the
workflow code, and imposed a pointless limitation of only allowing the use of the custom working directory system on
Windows runners.
Removing the unnecessary condition makes the workflow easier to understand and maintain, and makes it possible to
configure any job to use a custom working directory if necessary.
The build workflow produces builds for a range of target host architectures, including macOS Apple Silicon. This is done
by running a native build in a machine of the target architecture.
At the time the support for producing Apple Silicon builds was added to the workflow, use of GitHub-hosted Apple Silicon
runner machines was charged by the minute (while use of the other runners is free). In order to avoid excessive
expenses, the workflow was configured so that the Apple Silicon builds were only produced when absolutely necessary.
This was done by defining two sets of job matrix arrays, one for jobs using free runners, and the other for jobs using
paid runners. Due to the limitations of the GitHub Actions framework, it was necessary to use workflow environment
variables for this purpose.
Since that time, GitHub made free GitHub-hosted Apple Silicon runners available. When the workflow was adjusted to use
that runner, the configuration for the Apple Silicon build job was moved to the free runner job matrix array. The system
for supporting selective use of paid GitHub-hosted runners to produce builds was not removed at that time. This is
reasonable since it is possible the need will arise for using paid runners at some point in the future (e.g., only
legacy older versions of free macOS "Intel" runners are now provided and it is likely that even these will eventually be
phased out forcing us to use the paid runner to produce builds for that target). However, the environment variable for
the paid runner job matrix array data was removed. The absence of that variable made it very difficult to understand the
workflow code for the system.
For this reason, the environment variable is replaced, but empty of data. A comment is added to explain the reason for
this.
Document prerequisites of the Arduino CLI, LS, etc. tools when built
from a Git commitish.
---------
Signed-off-by: dankeboy36 <dankeboy36@gmail.com>
Co-authored-by: per1234 <accounts@perglass.com>
