{"id":7444,"date":"2025-10-10T20:43:18","date_gmt":"2025-10-10T20:43:18","guid":{"rendered":"https:\/\/robertjwallace.com\/?p=7444"},"modified":"2025-10-10T20:43:21","modified_gmt":"2025-10-10T20:43:21","slug":"the-trust-problem-why-you-cant-always-trust-the-software-you-run","status":"publish","type":"post","link":"https:\/\/robertjwallace.com\/es\/the-trust-problem-why-you-cant-always-trust-the-software-you-run\/","title":{"rendered":"The Trust Problem: Why You Can\u2019t Always Trust the Software You Run"},"content":{"rendered":"<p class=\"\">We rely on software every day, and we usually assume that if a major company releases a program, it must be safe. But there\u2019s a famous concept in computer science that shows exactly why that trust can be easily broken, even by the most well-meaning developers.<\/p>\n\n\n\n<p class=\"\">It all comes down to a fundamental question: How do you verify the tools that build the software?<\/p>\n\n\n\n<!--more-->\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">The Ultimate &#8220;Trusting Trust&#8221; Trick<\/h3>\n\n\n\n<p class=\"\">In 1984, computer pioneer Ken <strong>Thompson<\/strong> demonstrated a shocking theoretical attack involving a <strong>compiler<\/strong> <img decoding=\"async\" src=\"\" width=\"0\" height=\"0\"> (a program that translates human-readable code into computer-executable instructions). Thompson showed he could compromise the compiler&#8217;s binary executable\u2014the running program\u2014with two secret pieces of logic <img decoding=\"async\" src=\"\" width=\"0\" height=\"0\">:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li class=\"\"><strong>The Invisible Payload:<\/strong> A hidden feature that watches for specific code (like the system&#8217;s login program). When it sees that code, it automatically inserts a secret <strong>backdoor<\/strong> (like a master password) into the resulting compiled program <img decoding=\"async\" width=\"0\" height=\"0\" src=\"\">.<\/li>\n\n\n\n<li class=\"\"><strong>The Self-Replicator:<\/strong> A second feature that watches for the compiler&#8217;s own source code. When it sees itself being compiled, it automatically injects both the invisible payload and the self-replicator into the new compiler binary.<\/li>\n<\/ul>\n\n\n\n<p class=\"\">The genius of the hack is that the source code for the compiler remains perfectly clean. You could read every line of the compiler&#8217;s source code and never find the malicious instructions <img decoding=\"async\" src=\"\" width=\"0\" height=\"0\">.<\/p>\n\n\n\n<p class=\"\">This means:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li class=\"\"><strong>Audit Fails:<\/strong> A human reviewer looking at the source code sees nothing wrong <img decoding=\"async\" width=\"0\" height=\"0\" src=\"\">.<\/li>\n\n\n\n<li class=\"\"><strong>Perpetuation:<\/strong> Any new version of the compiler built with the infected one is automatically infected, passing the flaw down forever.<\/li>\n<\/ul>\n\n\n\n<p class=\"\">The trust chain is broken at the very foundation: you can&#8217;t trust the final program because you can&#8217;t trust the tool that built it.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">The Looming Threat of the Deep Supply Chain<\/h3>\n\n\n\n<p class=\"\">Thompson\u2019s concept reveals the danger of a <strong>software supply chain attack<\/strong> <img decoding=\"async\" src=\"\" width=\"0\" height=\"0\">. This threat has grown exponentially because modern programs are no longer built by one team from scratch <img decoding=\"async\" src=\"\" width=\"0\" height=\"0\">.<\/p>\n\n\n\n<p class=\"\">Today&#8217;s software is a tower of <strong>dependencies<\/strong>:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li class=\"\">A developer writes a little code.<\/li>\n\n\n\n<li class=\"\">They rely on hundreds or thousands of <strong>third-party libraries<\/strong> (small, reusable pieces of code) to handle tasks like data encryption, networking, or handling dates.<\/li>\n\n\n\n<li class=\"\">These third-party libraries often rely on other libraries, creating a <strong>deep dependency chain<\/strong> <img decoding=\"async\" width=\"0\" height=\"0\" src=\"\">.<\/li>\n\n\n\n<li class=\"\">All of this is processed by development tools (compilers, interpreters, build systems).<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Compromise at the Foundation<\/h4>\n\n\n\n<p class=\"\">An attacker doesn&#8217;t need to hack a major company directly. They just need to plant a flaw in one obscure, open-source library that is three or four levels deep in the dependency chain.<\/p>\n\n\n\n<p class=\"\">Real-world attacks have already shown the devastating results:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li class=\"\"><strong>SolarWinds (2020):<\/strong> Attackers injected malicious code into the build process of a trusted IT management program <img decoding=\"async\" width=\"0\" height=\"0\" src=\"\">. When thousands of corporate and government customers downloaded the legitimate update, they unknowingly installed the pre-compromised code, giving hackers widespread access.<\/li>\n\n\n\n<li class=\"\"><strong>Open-Source Backdoors:<\/strong> Hackers frequently compromise popular open-source code packages or create malicious &#8220;<strong>typosquatted<\/strong>&#8221; packages with similar names to legitimate ones, tricking developers into building the Trojan horse directly into their new software <img decoding=\"async\" width=\"0\" height=\"0\" src=\"\">.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Danger to Critical Infrastructure<\/h4>\n\n\n\n<p class=\"\">As society becomes a network of interconnected systems, the risk of a deep supply chain compromise is catastrophic. Compromising a single tool used by an engineering firm could inject undetectable flaws into:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li class=\"\">Power Grids<\/li>\n\n\n\n<li class=\"\">Water Treatment Facilities<\/li>\n\n\n\n<li class=\"\">Financial Trading Systems<\/li>\n\n\n\n<li class=\"\">Military Communications<\/li>\n<\/ul>\n\n\n\n<p class=\"\">A persistent, self-replicating flaw hidden deep in the toolchain could become a <strong>digital ticking time bomb<\/strong>.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">The Emerging Role of AI in the Trust Problem<\/h3>\n\n\n\n<p class=\"\">The rise of <strong>Artificial Intelligence (AI)<\/strong> tools in software development introduces a new and potentially more sophisticated vector for the &#8220;Trusting Trust&#8221; problem <img decoding=\"async\" src=\"\" width=\"0\" height=\"0\">.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">AI as the Invisible Injector<\/h2>\n\n\n\n<p class=\"\">AI coding assistants (like GitHub Copilot) are trained on massive amounts of code. An attacker could intentionally &#8220;poison&#8221; the training data with small, subtle patterns or rules . The AI model might then:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li class=\"\"><strong>Subtly Suggest Flawed Code:<\/strong> The AI could be prompted to generate code that is functionally correct but contains a tiny, hidden security flaw or an unintentional backdoor that is nearly impossible for a human reviewer to spot.<\/li>\n\n\n\n<li class=\"\"><strong>Utilize Hidden Instructions:<\/strong> Researchers have already shown that attackers can hide malicious instructions in files that the AI uses for context <img decoding=\"async\" width=\"0\" height=\"0\" src=\"\">. These instructions, often invisible to the human eye, direct the AI to inject a malicious script into the final code it generates.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Backdooring the AI Model Itself<\/h2>\n\n\n\n<p class=\"\">A more advanced threat involves planting a backdoor directly into the AI model&#8217;s learned data (its weights). This backdoored AI model could perform perfectly under normal testing, but if it detects a specific, rare &#8220;trigger&#8221; in the source code it&#8217;s reviewing, it could be forced to perform a malicious function, such as:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li class=\"\"><strong>Passing Malicious Code:<\/strong> Automatically flagging a known piece of malware as &#8220;clean&#8221; during a security review.<\/li>\n\n\n\n<li class=\"\"><strong>Generating Exploitable Code:<\/strong> Automatically inserting a specific vulnerability when prompted by a seemingly innocuous piece of code.<\/li>\n<\/ul>\n\n\n\n<p class=\"\">Just as Ken Thompson&#8217;s compiler was compromised without a trace in its source code, a modern AI model could be compromised without any visible fault in its training data or architecture, making the problem of trusting the tools more complicated than ever before <img decoding=\"async\" src=\"\" width=\"0\" height=\"0\">.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">Here is a list of sources relevant to the key concepts and examples mentioned in the blog post:<\/h3>\n\n\n\n<h2 class=\"wp-block-heading\">Core Concepts<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li class=\"\"><strong>Ken Thompson&#8217;s &#8220;Reflections on Trusting Trust&#8221;<\/strong> (The Ultimate &#8220;Trusting Trust&#8221; Trick):\n<ul class=\"wp-block-list\">\n<li class=\"\"><strong>Source:<\/strong> Thompson, Ken. &#8220;Reflections on Trusting Trust.&#8221; <em>Comunicaciones de la ACM<\/em>, Vol. 27, No. 8, pp. 761-763, August 1984.<\/li>\n\n\n\n<li class=\"\"><strong>Link:<\/strong> <em>This is his original Turing Award lecture where the concept was presented.<\/em> A widely cited version is available <a href=\"https:\/\/www.cs.cmu.edu\/~rdriley\/487\/papers\/Thompson_1984_ReflectionsonTrustingTrust.pdf\" target=\"_blank\" rel=\"noreferrer noopener\">aqu\u00ed<\/a>.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li class=\"\"><strong>General Software Supply Chain Attack Threat:<\/strong>\n<ul class=\"wp-block-list\">\n<li class=\"\"><strong>Source:<\/strong> National Institute of Standards and Technology (NIST) and various cybersecurity reports.<\/li>\n\n\n\n<li class=\"\"><strong>Link:<\/strong> <em>For a general overview of the increasing threat:<\/em> Searching for &#8220;software supply chain attack recent examples&#8221; or &#8220;deep dependency chain security&#8221; will yield numerous articles and industry white papers.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Real-World Attack Examples<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li class=\"\"><strong>SolarWinds (2020) Attack:<\/strong>\n<ul class=\"wp-block-list\">\n<li class=\"\"><strong>Source:<\/strong> Official reports and analysis from cybersecurity firms (FireEye\/Mandiant) and government agencies (CISA, ODNI).<\/li>\n\n\n\n<li class=\"\"><strong>Link:<\/strong> <em>Initial discovery and details from FireEye:<\/em> <a href=\"https:\/\/cloud.google.com\/blog\/topics\/threat-intelligence\/evasive-attacker-leverages-solarwinds-supply-chain-compromises-with-sunburst-backdoor\" target=\"_blank\" rel=\"noreferrer noopener\">Google Cloud Blog: SolarWinds Supply Chain Attack Uses SUNBURST Backdoor<\/a><\/li>\n\n\n\n<li class=\"\"><strong>Link:<\/strong> <em>Government\/Intelligence assessment:<\/em> <a href=\"https:\/\/www.dni.gov\/files\/NCSC\/documents\/SafeguardingOurFuture\/SolarWinds%20Orion%20Software%20Supply%20Chain%20Attack.pdf\" target=\"_blank\" rel=\"noreferrer noopener\">ODNI: SolarWinds Orion Software Supply Chain Attack<\/a><\/li>\n<\/ul>\n<\/li>\n\n\n\n<li class=\"\"><strong>Open-Source Backdoors and Typosquatting:<\/strong>\n<ul class=\"wp-block-list\">\n<li class=\"\"><strong>Source:<\/strong> Security firm vulnerability research and industry blogs.<\/li>\n\n\n\n<li class=\"\"><strong>Link:<\/strong> <em>Information on compromised popular open-source packages and typosquatting:<\/em> Search for &#8220;npm supply chain attack&#8221; or &#8220;malicious PyPI packages.&#8221; For example, <a href=\"https:\/\/www.sonatype.com\/resources\/vulnerability-timeline\" target=\"_blank\" rel=\"noreferrer noopener\">A History of Software Supply Chain Attacks &#8211; Sonatype<\/a> provides many examples.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">AI and the Trust Problem<\/h2>\n\n\n\n<p class=\"\"><strong>Link:<\/strong> <em>General overview of vulnerabilities in AI-generated code:<\/em> <a href=\"https:\/\/www.kaspersky.com\/blog\/vibe-coding-2025-risks\/54584\/\" target=\"_blank\" rel=\"noreferrer noopener\">Kaspersky: Security risks of vibe coding and LLM assistants for developers<\/a><\/p>\n\n\n\n<p class=\"\"><strong>AI Coding Assistant Security Risks (Poisoning, Backdoors):<\/strong><\/p>\n\n\n\n<p class=\"\"><strong>Source:<\/strong> Academic studies and threat research reports from security vendors.<\/p>\n\n\n\n<p class=\"\"><strong>Link:<\/strong> <em>Analysis of AI code assistant risks, including indirect prompt injection:<\/em> <a href=\"https:\/\/unit42.paloaltonetworks.com\/code-assistant-llms\/\" target=\"_blank\" rel=\"noreferrer noopener\">Palo Alto Networks Unit 42: The Risks of Code Assistant LLMs: Harmful Content, Misuse and Deception<\/a><\/p>\n\n\n\n<p class=\"\"><\/p>","protected":false},"excerpt":{"rendered":"<p>We rely on software every day, and we usually assume that if a major company releases a program, it must be safe. But there\u2019s a famous concept in computer science that shows exactly why that trust can be easily broken, even by the most well-meaning developers. It all comes down to a fundamental question: How &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/robertjwallace.com\/es\/the-trust-problem-why-you-cant-always-trust-the-software-you-run\/\" class=\"more-link\">Continuar leyendo<span class=\"screen-reader-text\"> &#8220;The Trust Problem: Why You Can\u2019t Always Trust the Software You Run&#8221;<\/span><\/a><\/p>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"nf_dc_page":"","_eb_attr":"","footnotes":""},"categories":[143],"tags":[],"class_list":["post-7444","post","type-post","status-publish","format-standard","hentry","category-computer-stuff"],"featured_image_src":null,"featured_image_src_square":null,"author_info":{"display_name":"Bob","author_link":"https:\/\/robertjwallace.com\/es\/author\/admin\/"},"_links":{"self":[{"href":"https:\/\/robertjwallace.com\/es\/wp-json\/wp\/v2\/posts\/7444","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/robertjwallace.com\/es\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/robertjwallace.com\/es\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/robertjwallace.com\/es\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/robertjwallace.com\/es\/wp-json\/wp\/v2\/comments?post=7444"}],"version-history":[{"count":2,"href":"https:\/\/robertjwallace.com\/es\/wp-json\/wp\/v2\/posts\/7444\/revisions"}],"predecessor-version":[{"id":7446,"href":"https:\/\/robertjwallace.com\/es\/wp-json\/wp\/v2\/posts\/7444\/revisions\/7446"}],"wp:attachment":[{"href":"https:\/\/robertjwallace.com\/es\/wp-json\/wp\/v2\/media?parent=7444"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/robertjwallace.com\/es\/wp-json\/wp\/v2\/categories?post=7444"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/robertjwallace.com\/es\/wp-json\/wp\/v2\/tags?post=7444"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}