Add fossa to pipeline
This adds FOSSA to the GeoSpatial pipeline.
Beyond the simple inclusion of the fossa-maven.yml
file, the project also needed a .fossa.yml
file to define the modules to scan. I use a basic fossa init
call to generate this file. Note -- we're still using fossa 1.x, which is why we need the fossa configuration file. The next version of fossa (3.x -- there was no 2.x) no longer needs this file.
What the steps do
fossa-analyze
The fossa-analyze
job collects the dependency data for the project, uploads it to FOSSA's servers for analysis, then passes or fails based on whether any licensing issues were found. Since this is defined in the fossa-maven.yml
file, there's a little bit of extra maven-based setup that it does first. But when it actually gets to running the analysis, everything in the .fossa-yml
is used.
Since your other code is NPM based, and NPM doesn't need as much configuration, this one job is able to handle both. Which is pretty lucky. The standard fossa jobs aren't designed to be used in multi-language projects, but if it works for you, that's great.
If you see a failure in this job that results from a licensing error count > 0, your only recourse is to tag me to look into it. They have to be resolved in the FOSSA control panel, and I'm the only one with an account there. It's a frustrating bottleneck, but needs to be that way for various reasons. If it fails because of a build error (like, your java code doesn't compile), then you may be able to manage it yourself. Those kinds of failures are almost always due to a configuration parameter that isn't being sent along correctly -- since the pipeline already passed the compile-and-unit-test
phase, we know that the code can be built.
fossa-check-notice
The fossa-check-notice
file will generate a fresh attribution file (aka NOTICE
) file, and compare it against the one committed in the repository. These are required by most open source licenses (not NOTICE files specifically, but attribution in some form). If there's a difference, this step will fail and ask you to review. The differences are shown in the job output, and the generated (or cached) NOTICE is available for you to download and commit.
One note -- FOSSA can be a bit erratic. Sometimes it drops packages for no reason, but more often it changes the way it spells the package name or source code location. I've taken steps to try to smooth this over. That's what the "cache" part is. If I detect that the dependencies haven't changed at all, then I will compare your committed NOTICE against a previous attribution generation. Caches can only be held if dependencies don't change (even one small version change of a deep dependency invalidates the cache), and they can only be used for a week before needing to be refreshed. I'm always working to improve this process to have less notice failures, but you'll undoubtedly encounter some.
I put some instructions in the log file about how to download the artifacts and commit them. You don't have to use those exact commands if you want to go about it another way. The basic idea is to look at the job artifacts for fossa-analyze
, download the generated-NOTICE
or cached-NOTICE
file, then commit it as NOTICE
.
I don't automatically update the NOTICE
because I want a human in the loop. When you're doing this, you should double check that the changes are real -- you actually are using these new dependencies, or have stopped using ones it wants to remove. If they look wrong, flag me. In particular, if your attributes are mostly / completely wiped out in a check-notice, then there's probably a scan failure that isn't detecting things right. This hasn't happened in a long time, but it's worth keeping an eye out for.
License Failures Detected
This is a list of licensing failures that FOSSA detected on the first run. These should be addressed soon. Either as part of merging this MR, or shortly thereafter in separate MRs.
-
flora-sql-parser, Closes #250 (closed), see comments -
wicket, #251 (closed) -
false positives (diff, turfjs) -- further analysis determined these were incorrectly categorized by FOSSA and are not actual license concerns. I've corrected the FOSSA databases, so these are no longer flagged.