Slashdot is powered by your submissions, so send in your scoop

 



Forgot your password?
typodupeerror
Programming Security

Secure Software Supply Chains, Urges Former Go Lead Russ Cox (acm.org) 16

Writing in Communications of the ACM, former Go tech lead Russ Cox warns we need to keep improving defenses of software supply chains, highlighting "promising approaches that should be more widely used" and "areas where more work is needed." There are important steps we can take today, such as adopting software signatures in some form, making sure to scan for known vulnerabilities regularly, and being ready to update and redeploy software when critical new vulnerabilities are found. More development should be shifted to safer languages that make vulnerabilities and attacks less likely. We also need to find ways to fund open source development to make it less susceptible to takeover by the mere offer of free help. Relatively small investments in OpenSSL and XZ development could have prevented both the Heartbleed vulnerability and the XZ attack.
Some highlights from the 5,000-word article:
  • Make Builds Reproducible. "The Reproducible Builds project aims to raise awareness of reproducible builds generally, as well as building tools to help progress toward complete reproducibility for all Linux software. The Go project recently arranged for Go itself to be completely reproducible given only the source code... A build for a given target produces the same distribution bits whether you build on Linux or Windows or Mac, whether the build host is X86 or ARM, and so on. Strong reproducibility makes it possible for others to easily verify that the binaries posted for download match the source code..."
  • Prevent Vulnerabilities. "The most secure software dependencies are the ones not used in the first place: Every dependency adds risk... Another good way to prevent vulnerabilities is to use safer programming languages that remove error-prone language features or make them needed less often..."
  • Authenticate Software. ("Cryptographic signatures make it impossible to nefariously alter code between signing and verifying. The only problem left is key distribution...") "The Go checksum database is a real-world example of this approach that protects millions of Go developers. The database holds the SHA256 checksum of every version of every public Go module..."
  • Fund Open Source. [Cox first cites the XKCD cartoon "Dependencies," calling it "a disturbingly accurate assessment of the situation..."] "The XZ attack is the clearest possible demonstration that the problem is not fixed. It was enabled as much by underfunding of open source as by any technical detail."

The article also emphasized the importance of finding and fixing vulnerabilities quickly, arguing that software attacks must be made more difficult and expensive.

"We use source code downloaded from strangers on the Internet in our most critical applications; almost no one is checking the code.... We all have more work to do."


Secure Software Supply Chains, Urges Former Go Lead Russ Cox

Comments Filter:
  • by fuzzyfuzzyfungus ( 1223518 ) on Sunday September 21, 2025 @12:45PM (#65674466) Journal
    This seems like the sort of advice that is going to be exceptionally hard to get followed because it's mostly so dull.

    There can be some interesting futzing in principle to keep unnecessary sources of variation from getting folded into build artifacts, normally followed by less-interesting making of those change in practice across a zillion projects; and basically anything involving signing should at least be carefully copying the homework of proper heavyweight cryptographers; but most of the advice is of the "fix your shit" and "yes, actually, have 10 people, ideally across multiple orgs, despite the fact that you can get it for free by pretending that the random person in Nebraska won't make mistakes, get coopted by an intelligence agency, quit to find a hobby that doesn't involve getting yelled at on the internet for no money, or die" flavor; which is absolutely stuff you should do; but the sort of deeply unsexy spadework that doesn't have magic bullet vendors lobbying for it to get paid for.
    • This is why they're also recommending paying for open source development.

      This sort of payment would generally be through direct employment or funding a nonprofit like ASF. In either case, someone can actually be expected to make reproducible builds, address vulnerabilities, and handle build signing.

  • From the article: "The only problem left is key distribution: The verifier must know who should have signed the code. [...] To the extent that questions of identity can be solved, having authors sign their software can provide even stronger guarantees." It goes on to describe how Debian and Go package repositories include the expected hash value of a package, so that package downloading tools can reject a package that has been replaced.

    However, the approach used by Debian to verify developers' identity, that of new developers physically meeting existing trusted developers at key signing parties to exchange OpenPGP public keys, doesn't scale very well. A lot of contributors are disconnected from the strongly connected set of the web of trust because they cannot travel to key signing parties. This can be because of cost, work or child care scheduling, regulatory restrictions related to geopolitics, or regulatory restrictions related to public health (most recently during 2020-2021). These disconnected contributors must forever rely on the bottleneck of "sponsors" (trusted developers who forward packages from the maintainer to the distribution) to get their work into a distribution.

    And sponsors are indeed a bottleneck. From the article: "And then you need to be ready to update to a fixed version of that dependency." When a package's upstream maintainer releases an updated version of a package, the package's sponsor in a particular distribution may be too busy with other tasks to handle it the same day. This can mean that there is no available labor to forward the update to the rolling distribution and backport the fix to the version of the package in a stable distribution.

    • The verifier must know who should have signed the code.

      The problem is that is never knowable, even under best case scenarios. All it takes is one compromised key and you've got people installing something they shouldn't, believing the "padlock" that it's "secure". Even if that doesn't happen, if the attacker can get a code signature that is "trusted by default", you're still screwed. It doesn't matter whether that's because getting a trusted signature is too easy (Let's Encrypt) or because of some bug in an actually trusted program (OS library signed by the ma

  • From the article:

    The Go project recently arranged for Go itself to be completely reproducible given only the source code, meaning that although a build needs some computer running some operating system and some earlier Go toolchain, none of those choices matters."

    [...]

    The Multics review is famous for pointing out the possibility of adding a back door to a compiler to insert back doors in critical system programs during compilation [...]. Reading the report inspired Ken Thompson to implement exactly that att

  • Prevent Vulnerabilities. "The most secure software dependencies are the ones not used in the first place: Every dependency adds risk... Another good way to prevent vulnerabilities is to use safer programming languages that remove error-prone language features or make them needed less often..."

    The problem isn't principally language constructs. It is that large amounts of code are required for modern software. As an exercise, just try to develop your own code to support https fully and correctly - including common add-ons as Digest and OAuth security - you must depend upon a library to do that. There are many more protocols that are every bit as complicated (try reading a 3d file in a browser, for example).

    I think the best thing a language can do is provide a rich set of first-party standard libra

    • As an exercise, just try to develop your own code to support https fully and correctly - including common add-ons as Digest and OAuth security - you must depend upon a library to do that.

      OAuth isn't an HTTP extension, it's an OSI application level protocol. Also, of all the protocols you could have picked, OAuth is a really simple one. You could code a custom implementation in an afternoon. HTTP isn't even so bad, even with the HTTP/2 modifications, you could definitely do it yourself, depending on your reading comprehension ability (reading code standards is not a skill every programmer has).

      The built-in functions for Javascript (and their typical runtime environment - the browser) are minimal leading to a lot of dependence upon 3rd-party libraries.

      Ever since most of JQuery functionality got added to the Javascript standard library, you mostly d

  • I still don't understand the lack of discussion about the nightmarish insecurity of Docker repos and Powershell nuget gallery. People are happily downloading and executing all sorts of random shit, as root, without a second thought.

  • I feel like he's a car salesman trying to sell us how revolutionary automatic transmission is. Everything he described is well known and practiced in the Java community before Go was invented...and Java runs A LOT FASTER than Go....which is not relevant, but why I hate that language so much. With Rust being superior, it's garbage forced on us that should go the way of COBOL. You want reliability, "reproducibility" and performance...Java is far easier to work with, has a larger talent pool, and is much bet
    • The difficulty with reproducible builds in Java is the timestamps. It's a solved problem now, but I don't think it was 20 years ago.
  • by Mirnotoriety ( 10462951 ) on Sunday September 21, 2025 @03:00PM (#65674616)
    a. Only download and compile from primary sources.

    b. Have checksums stored on a separate server.

    c. Have the developers not click on unsafe URLs they receive in emails.

    ps: In this day and age, isn't it possible to make digitally signed emails de rigueur.
  • As a maintenance programmer for most of my career, non-reproducible builds were the bane of my existence. Object files with embedded time stamps and source paths, changes in link order, which module got done compiling first when doing multiple modules in parallel... Hell, even uninitialized padding bytes to align to word boundaries could change from build to build. We were using C and gcc, so it might be fair to say that was our problem. I'm not even sure it's possible for that toolchain to produce identic
  • If you want to build from nothing but source code, how do you bootstrap the build? The first thing you'll need is to create a compiler, probably from a shell script? How do you know your shell isn't compromised?

    • Start with stage0 [github.com] (whose binary seed is about 1 KiB) and GNU Mes [gnu.org]. Use mescc to build tinycc, then GCC 2.95, then GCC 4.7, then fairly modern GCC, and then use mrustc [github.com] to build some version of Rust. The time-consuming part is that each version of the Rust toolchain uses fairly new features in the Rust language, so yes, you'll probably have to build the world a couple dozen times starting with the most recent version supported by mrustc.

  • There was an attempted attack about two years back. I seem to recall the attackers had to back up a few repos to get their code injected and undetectable. IIRC they first (successfully) attacked the tools to build the final project.

    Anyone remember the details?

I think there's a world market for about five computers. -- attr. Thomas J. Watson (Chairman of the Board, IBM), 1943

Working...