Block CISO Discusses AI Security Challenges and Prompt Injection Risks
- Securityish
- AI & Future Technology
Quick Summary
Block’s Chief Information Security Officer, James Nettesheim, highlighted security challenges associated with their AI agent, Goose, which is used by 12,000 employees. The company recently red-teamed Goose, discovering vulnerabilities such as prompt injection that allowed malware to infect an employee’s laptop. This incident underscores the importance of applying least-privilege access and continuous security testing in AI systems.
The Securityish Brief
Block’s CISO, James Nettesheim, emphasized the need for AI systems to be not only effective but also secure, drawing parallels to self-driving cars. The company has developed Goose, an open-source AI agent utilized by its 12,000 employees, which connects to various systems including Google accounts and Square payments. Recently, Block conducted a red team exercise on Goose, where they successfully executed a prompt injection attack that led to malware being installed on an employee’s laptop.
Prompt injection occurs when malicious instructions are embedded in prompts that the AI executes. In this case, the red team used a combination of phishing emails and prompt injection techniques to trick a developer into executing a poisoned recipe, which ultimately downloaded an information-stealing malware. This incident highlights the potential risks associated with AI agents and the necessity for robust security measures.
To mitigate these risks, Block is implementing features such as recipe install warnings and alerts for suspicious Unicode characters. These measures aim to enhance transparency and user awareness regarding the execution of new workflows. Additionally, the company is exploring adversarial AI techniques to improve security by using AI to monitor and validate the prompts being processed by Goose.
Implications for Security Practices
This incident serves as a critical reminder for organizations to adopt least-privilege access principles for both human users and AI systems. Ensuring that employees and AI agents only have access to the data necessary for their tasks can significantly reduce the risk of data breaches and unauthorized access.
Organizations should also prioritize regular security testing and red teaming to identify vulnerabilities in their systems. By simulating attacks, companies can better understand potential weaknesses and implement necessary safeguards before they can be exploited by malicious actors.
As AI technology continues to evolve, organizations must remain vigilant and proactive in their security practices. This includes monitoring for new types of attacks, such as prompt injection, and adapting security measures accordingly to protect sensitive data and maintain user trust.
Key Takeaways
- Implement least-privilege access for both employees and AI systems to minimize data exposure.
- Conduct regular penetration testing and red teaming exercises to identify vulnerabilities in AI applications.
- Educate employees about the risks of phishing and prompt injection attacks to enhance awareness.
- Monitor and review AI workflows for suspicious activities or unauthorized changes.
- Stay updated on emerging AI security threats and adapt security measures accordingly.
Key Terms & Concepts
- Prompt Injection: In this article, prompt injection refers to a technique where malicious instructions are embedded in prompts that an AI executes.
- Least-Privilege Access: Least-privilege access is a security principle that ensures users and systems only have access to the data necessary for their roles.
- Adversarial AI: Adversarial AI involves using artificial intelligence to attack or test the security of other AI systems.
Read the original article here: https://go.theregister.com/feed/www.theregister.com/2026/01/12/block_ai_agent_goose/
Your 5-Minute Securityish Brief
A weekly digest of cybersecurity news, phishing alerts, privacy tips, and emerging threats, simplified so anyone can understand what matters and why.
Securityish
Securityish explains cybersecurity, scams, data breaches, and privacy risks in simple language so you know what’s happening and how to protect yourself.
© 2026 Securityish
