Anatomy of the Lightning PyPI compromise
By Beatriz Costa
Safety advisory: SFTY-20260430-22532 — affected versions: lightning 2.6.2, lightning 2.6.3. Block these versions immediately; rotate any credentials reachable from a host that imported them.
TL; DR. Two malicious versions of the lightning PyPI package versions 2.6.2 and 2.6.3 were published on April 30, 2026, with a credential-stealing payload that executes on import. PyPI quarantined the malicious releases within hours, and public reporting at the time described the project itself as quarantined. Forensic analysis indicates that the attacker bypassed Lightning AI's CI pipeline: there is no GitHub release for either version, the wheel was built on a different machine than the legitimate January release (Python 3.14.3 vs 3.9.25, setuptools 82.0.1 vs 80.10.2), and only the umbrella package was pushed. The malicious payload is a 4-file change set: two added runtime files plus a 19-line __init__.pylauncher and a 3-line __setup__.py packaging directive that ensures the runtime ships in the wheel. The available metadata is consistent with CI bypass and the use of valid publishing credentials outside the normal release path; it does not prove whether the credentials originated from a PyPI token leak, a maintainer-account compromise, or a workstation compromise.
If you are unsure whether your team has installed these versions, Safety Endpoint Security will detect them for you, and they can be rolled out to your entire team in minutes.
The incident in a nutshell
lightning is the umbrella distribution for the PyTorch Lightning ecosystem where over 11 million PyPI downloads per month. It depends on ecosystem packages such as pytorch-lightning and lightning-fabric; those sibling packages do not declare a dependency back on lightning. On April 30, 2026, versions 2.6.2 and 2.6.3 were published, carrying a payload that runs on import: it spawns a daemon thread, downloads the Bun JavaScript runtime from GitHub, and executes an 11 MB obfuscated JS payload.
First-layer static analysis of the JavaScript confirms code paths for handling npm publish tokens, GitHub tokens with both API-based exfiltration and commit-write behavior, AWS/GCP/Azure credential collection, and cloud secret manager SDK calls. Wallet, VPN-config, and shell-history targeting documented by external researchers running the payload in instrumented sandboxes is not directly observable in the first-layer string table. PyPI quarantined the malicious releases, and public reporting noted that the project was quarantined.
Public reporting has linked the attack to the Mini Shai-Hulud campaign via technical-signature overlap with the litellm, Bitwarden CLI, Checkmarx KICS, and SAP CAP-js compromises of the past several weeks; the same reporting also notes a TeamPCP-branded self-claim posted via Tor onion link during the live window, while flagging that the attribution has not been independently verified and may be opportunistic branding or a false flag. The Lightning AI team has acknowledged the incident and confirmed it as a compromise; the root cause is still under investigation.
The malicious payload
The wheel-vs-wheel comparison from clean lightning-2.6.1-py3-none-any.whl (Jan 30) to malicious lightning-2.6.3-py3-none-any.whl(Apr 30) shows a larger raw diff than the malicious payload itself, because three months of legitimate Lightning AI development sits between those releases. Separating the two:
Everything else in the wheel diff: 28 modified files, including version.info, Fabric/PyTorch internals, callbacks, and changelogs, are legitimate development carried over from the dev branch the attacker built on top of.
The __setup__.py change is worth pausing on. Without that three-line edit, setuptools would have ignored the _runtime/directory entirely — non-Python files in non-package subdirectories don't ship in a wheel by default. The attacker explicitly added package_data: {"lightning": ["_runtime/*"]} to the package configuration to ensure their payload actually got bundled.
Combined with the Popen/DEVNULL/daemon-thread style of the launcher, this reads as someone who has shipped real Python code and understands the packaging toolchain.
The injection vector
The diff inserts a 19-line block near the top of lightning/__init__.py, immediately after the existing import logging line:
Inserting at the top rather than just appending at the bottom means the daemon thread launches before any of the legitimate Lightning module setup runs. Even if a later legitimate import raises, the harvester is already running.
start.py is a stdlib-only Python bootstrapper (98 lines, 2,871 bytes) that downloads Bun 1.3.13 into _runtime/.bun/ inside the installed lightning package directory SCRIPT_DIR / ".bun" in the code and uses it to execute router_runtime.js from the same directory. Because the binary is cached alongside the bootstrapper rather than in a tempdir, subsequent imports skip the download step entirely; this is also a strong post-compromise indicator (defenders should grep their site-packages trees for lightning/_runtime/.bun/).
The launcher is daemonized so the parent Python process can exit cleanly without waiting on the harvester, and stdout/stderr are routed to /dev/null so a noisy import doesn't tip off a developer running an interactive python -c "import lightning".
A small detail between 2.6.2 and 2.6.3: start.py shrank by 595 bytes between the two releases, with all of the removed content being debug print() calls and terminal-mapped subprocess stdout/stderr. 2.6.3 strips every print and explicitly suppresses child output. Within roughly an hour. That cadence reads as a hands-on operator iterating on observability after publishing 2.6.2: not a fully automated worm.
How it bypassed Lightning AI's CI
The legitimate 2.6.1 release on PyPI is associated on the GitHub side with a release tag at commit 655a3a9, a merged PR that triggered a GitHub Actions workflow, a 3-minute CI run producing 4 artifacts, and PyPI uploads for pytorch_lightning-2.6.1, lightning_fabric-2.6.1, and lightning-2.6.1 clustered within 5 seconds of each other.
The pipeline does not use Trusted Publishing. Both the clean lightning-2.6.1 and the malicious lightning-2.6.3 lack any PEP 740 provenance attestation, and Safety's ingestion pipeline records both releases as uploaded via twine/6.2.0. The publishing identity is a long-lived PyPI API token used by twine from the GitHub Actions runner.
The malicious releases didn't go through that pipeline. The Lightning-AI/pytorch-lightning releases page jumps directly from Lightning v2.6.1 (Jan 30) to Lightning v2.6.0 (Nov 28); there is no 2.6.2 release, no 2.6.3 release, no corresponding tag, and no associated CI workflow run. The artifacts simply appeared on PyPI on April 30. Three corroborating fingerprints:
Generator: setuptools (82.0.1) in their WHEEL metadata; the legitimate Jan 30 batch was built with setuptools (80.10.2). Different build environment.CPython/3.14.3; the legitimate release reports CPython/3.9.25. Different Python.lightning was pushed neither; pytorch-lightning nor lightning-fabric got a 2.6.2 or 2.6.3 release. The legitimate workflow publishes all three from the same monorepo run within seconds. An attacker who had compromised the workflow itself would have shipped all three.The available metadata is consistent with CI bypass and the use of valid publishing credentials, both of which operate outside the normal release path. It does not prove whether the credential came from a PyPI token leak, a compromised maintainer account, or a workstation compromise. Lightning AI has not publicly disclosed the root cause.
Why only lightning got hit, not its siblings
The asymmetric impact pattern is itself diagnostic. The legitimate CI publishes all three monorepo packages in a single run; only the umbrella was pushed.
Two readings, both plausible: the credential was scoped to lightning only (PyPI tokens support per-project scoping), or the attacker had broader access but prioritized the umbrella for blast-radius reasons. The third logical possibility: broad access but interrupted by detection is unlikely given that 2.6.3 was uploaded after 2.6.2, indicating a working iteration loop.
Lightning AI maintains projects across multiple repositories with non-uniform publishing models. Safety verified each ecosystem package's Trusted Publishing status during analysis by querying PyPI's integrity/provenance endpoint at https://pypi.org/integrity/<pkg>/<ver>/<file>/provenance an HTTP 200 response with a Sigstore-signed attestation bundle confirms PEP 740 publication; a 404 confirms no attestation exists. We checked the latest release of every ecosystem package, plus a positive control (pip 26.1, which returned 200 as expected):
torchmetrics is the standout: its 1.9.0 release includes a verifiable PEP 740 attestation bundle that binds the artifact to commit 700261d3683ba77517e9650043f62a8313027437 of Lightning-AI/torchmetrics, generated by the publish-pkg.yml workflow. The bundle is signed by Sigstore and recorded in the public Rekor transparency log, with the originating GitHub Actions run at actions/runs/22866646546.
What this comparison shows is more limited than a clean "TP saved the day" story. Six of seven packages still use the long-lived-token model, and six of seven have survived. The pattern is consistent with token scoping, specifically bounding the blast radius for lightning.
What torchmetrics adds is that even if the attacker had held a broader-scoped credential, it would still have been protected; its publishing identity isn't a long-lived token at all. That's a structural guarantee, not a hygiene-dependent one.
The GitHub side: pl-ghost
Within minutes of the malicious versions being flagged on PyPI, a community member opened issue #21689 in Lightning-AI/pytorch-lightning, reporting the hidden execution chain. An external security researcher then opened a follow-up issue to escalate, which was closed within roughly one minute by an account named pl-ghost, with "PyTorchLightning & Grid.ai" per its public bio, and replied with a "SILENCE DEVELOPER" meme as the closing comment.
Public reports and the community issue history indicate suspicious pl-ghost activity consistent with compromise of either the bot's credentials or a maintainer session with admin scope. The exact account or session compromise path has not been publicly confirmed. The practical implication: the compromise extends beyond PyPI publishing to the project's GitHub presence, meaning a maintainer who realizes their PyPI token is compromised and tries to alert users via GitHub may find that channel denies them a voice as well.
What's still uncertain
pl-ghost is the specifically hijacked credential, or whether an attacker with access to a maintainer's session is operating it. Behavior observed; root cause undisclosed.Campaign context
The Lightning compromise sits inside a sustained string of supply-chain attacks across a roughly five-week window:
Public reporting on the campaign is divided on attribution: some coverage commits to a TeamPCP attribution citing Tor onion self-claims posted during the live windows; other coverage uses "Mini Shai-Hulud" as a technique-cluster label without attributing to a specific group.
We use the technique-cluster framing in this post because it does not depend on unverified self-claims, and the technical-signature linkage between incidents is a more durable observation regardless of how the operator is labeled.
Three patterns hold across the campaign. The point of compromise is consistently the maintainer's credential surface, not the codebase or repository: no attacker-authored commits appear in any of these projects' main branches
The execution mechanism is consistently auto-executed on import or install — __init__.py injection, .pth files, preinstall/postinstall hooks. The propagation logic is consistently lateral via the victim's own credentials, where npm packages get backdoored on the developer's machine using stolen tokens.
Indicators of compromise
Defenders sweeping for cross-campaign signs of compromise on hosts that imported the malicious package should additionally check for indicators reported elsewhere in the broader Mini Shai-Hulud campaign: GitHub commits authored as "Claude Code" (impersonation), repository branches named "shai-hulud", workflow files at .github/workflows/discussion.yaml or shai-hulud-workflow.yml, self-hosted GitHub runners registered as "SHA1HULUD", and commit messages of the form OhNoWhatsGoingOnWithGitHub:[Base64].
What to do
If you imported lightning 2.6.2 or 2.6.3 on any host during the live window, treat that host as compromised: rotate every credential reachable from it (GitHub PATs and SSH keys, npm tokens, AWS/GCP/Azure access keys, environment-variable secrets, package-registry credentials), audit recent activity on those credentials, and rebuild the host. Pin to lightning==2.6.1 in your dependency files until Lightning AI confirms credential rotation and resumes publishing.
If your team is evaluating its supply-chain posture in light of this incident, reach out to Safety — or get started immediately with pip install safety and safety scan.