Rendered at 18:50:29 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
cedws 3 hours ago [-]
A new chip solves nothing. Nobody wants to hear this but there is no solution for the security risks posed by agents today. You can put it in a sandbox, it doesn't make a difference, for it to be useful it inherently needs wide, unattended access. Put a human in the loop and you just end up bottlenecking it and throwing away any purported productivity gains. Auto mode doesn't matter either, it's trivial to trick and for the agent to break out.
nicce 2 hours ago [-]
> Put a human in the loop and you just end up bottlenecking it and throwing away any purported productivity gains. Auto mode doesn't matter either, it's trivial to trick and for the agent to break out.
Productivity gains are still enormous compared to what we used to do before agents. But, I know that people don't want to stop there.
egeozcan 2 hours ago [-]
Humans can also be tricked by the agents.
Humans can be tricked by humans too but humans care about their reputation in their communities, and at least fear from punishment.
paimapi 2 hours ago [-]
right, the solution here is not a hyper-capitalist race-to-the-bottom-of-devaluing-labor. it's recognizing discretion and diligence are things still required for work to be of a certain quality
bob1029 3 hours ago [-]
I feel like we are missing many shades of grey in the middle.
Semi-automation (human in the loop) can still result in a dramatic uplift in productivity. You can't run a combine harvester 100% autonomous but that doesn't stop anyone from trying to get as close to that limit as possible.
inetknght 2 hours ago [-]
> You can't run a combine harvester 100% autonomous
I'm curious why you think that.
theoreticalmal 2 hours ago [-]
Probably repair, refuel, what happens in a tornado. There’s an infinite amount of complexity in the world and a finite amount of computation
bob1029 2 hours ago [-]
Many forms of maintenance cannot be automated. Especially break fix maintenance.
mschuster91 2 hours ago [-]
Oh you absolutely can run them autonomously on the field. You only need a human these days to refuel them.
Precision Agriculture stuff is utterly crazy these days, other than fuel the remaining staff is the only thing left where you can get efficiency improvements - and at the scale of modern megafarms, even small percentages add up to a ton of money.
parsimo2010 2 hours ago [-]
Agreed- this is the same problem we have with trusted admins or devs who have elevated privileges on their networks. We have to trust that the admins won't use their power to steal company secrets or misuse company resources. If you don't trust the admins, then they can't fix things on your network and there is no point in having them.
If you want an agent to act on its own, like pushing to a git repo, managing dependencies, building and testing, etc., then you have to trust it as much as any other privileged user.
If you don't want to trust it, then you're just forcing yourself into the reverse centaur role, where the agent edits some code, but then has to stop and ask you to push the changes or build the software again and run the unit tests.
I suppose there is a principled way of doing things like "I trust you do do basic commits but I will handle merge conflicts" and "you can build modules in this directory but you can't build outside of it" but this is just a lot of effort that most orgs won't bother with.
DougN7 2 hours ago [-]
Even then if the agent goes rogue and decides to do the merges you can’t stop it if it has any kind of access. This goes back to the OP’s point - agents can’t be 100% constrained.
parsimo2010 2 hours ago [-]
You can absolutely run an agent as a limited-privilege user that only has write privileges for specific files and only has execute privileges for certain files. If it is running as a limited-privilege user it can work on code in it's own copy of the repo and make commits and send pull requests, but it can't do the merge. The problem is that nobody wants to go through the effort to set up all these permissions and nobody wants to take the time to review everything and perform all the manual actions.
cedws 2 hours ago [-]
Some shops are now generating tens or even hundreds of PRs a day with relatively little involvement. That volume is simply beyond what anyone can reasonably review.
la6479 2 hours ago [-]
Neither can be humans.
mixedbit 2 hours ago [-]
An agent doesn't inherently need wide access to be useful. The most popular application for agents today is writing code. A coding agent needs write access to the source code and read/execute access to tools needed to build and test the code, but not much more. There is little added utility from giving coding agent access to things like ssh keys.
cedws 2 hours ago [-]
If you're using agents to purely generate code with absolutely no way to reach the outside world, not even to fetch docs or dependencies, then sure the risks can be quite low. I haven't heard of anyone doing this though, and it would be incredibly challenging to make work given how much tooling needs to fetch from remote sources.
__MatrixMan__ 2 hours ago [-]
If your project truly depends on those things, they should be declared dependencies. Presumably you have some tool for injecting such things into a shell that the agent can use (I use nix for this). So if you run the agent from that shell, it has what it needs. If the shell doesn't have what it needs, that's a bug which the agent can fix by declaring new dependencies, but you have to relaunch the agent in the updated shell--so there's your opportunity to weigh in on whether the new resources are appropriate.
The benefits of being persnickety about precisely defined dependencies have outweighed the headaches since long before agents came on the scene. Agents have just made it even more important to do so, because if you let them fetch things all willy nilly like you'll have "works on my machine" problems at a much greater rate than was previously possible.
mixedbit 2 hours ago [-]
In cases where you need agents to fetch data from any remote source, sandboxing is still very much useful. Why give access to your ssh keys to network reaching agents?
Look at websites: websites are able to fetch code from any remote URL, yet browsers heavily use sandboxing to ensure that if fetched code turns out to be malicious, the users local files, cookies, etc are not exposed.
cedws 1 hours ago [-]
I'm afraid you're not thinking about this creatively enough, this topic is so much deeper applying a chroot or something and praying everything will be fine. So you give your agent internet access, OK what else does it have access to? Just read only access to your repo? The repo can be exfiltrated. Egress proxy only allows egress to GitHub? Repo can still be exfiltrated via GitHub. If the agent is poisoned (via prompt injection), it can tricked into searching for ways to escape.
For an agent to go rogue it doesn't even need to be directly able to access the internet. It just takes something to poison the context in the 'clean room' environment it operates, and if that poisoning manages to get a foothold, it can go dormant and hide like a virus. This kind of horrifying thing is going to happen on a large scale sooner or later.
ramoz 2 hours ago [-]
> but not much more
This is no longer true. Everyday I need my agents to access other repos, search the web, experiment/prototype, and deploy + integrate across other things.
CoolestBeans 2 hours ago [-]
The hypothesis I've had in my head since OpenClaw has been the following and I haven't seen contradictory evidence yet. Agents have a fundamental unresolvable tension between usefulness, safety, alignment, and accuracy. You have to restrict access to ensure an agent acts safely because alignment and accuracy cannot be perfect. But restricting access makes the agent less useful. You can play with the sliding scale and get more and more granular with access restrictions but at some point you need to draw some line. And then finally, even access restrictions cannot be made perfect, so improvements to model accuracy without corresponding improvements to alignment make detailed access controls less useful.
In other words, better models need blunter access controls which negates whatever improvement in utility they provide.
Matl 2 hours ago [-]
> a new chip solves nothing
It does allow Nvidia to sell more chips. This is no genuine attempt to solve anything, imo.
__MatrixMan__ 2 hours ago [-]
I don't see why it needs wide unattended access. There's no getting around spending some human time on expressing your wishes and constraints, but we have choices about what form that takes. Markdown files and wide access seems to work, but so does custom handcuffs for each job. You just have to shift your guidance out of documentation and into interactive help, error messages, or other facets of the handcuffs (e.g. a custom CLI for this task which is the only way for the agent to act outside of its sandbox).
johnsmith1840 3 hours ago [-]
"Inherently needs wide unattended access"
And what if you could? What if you could give a space secure enough it could have direct control over your bank account. It may do something dumb but it's boundaries are beyond the agent.
It could use your routing number and run your gmail without risk of abusing the routing number.
jagraff 2 hours ago [-]
How would it have access to my routing number and gmail without the risk of sharing my routing number over gmail?
johnsmith1840 1 hours ago [-]
Just assume it's possible, how interesting is it to you?
jagraff 55 minutes ago [-]
Oh I think I misread your comment slightly; I would not be interested in an agent that could do something dumb with my routing number, but if somehow there was an agent that I trusted as much as, eg, the payroll department at my employer, I would absolutely want and use that agent; I would love to have an agent that can handle all of the boring parts of my life such as paying bills, scheduling maintenance, dealing with bureaucracy, etc.
johnsmith1840 43 minutes ago [-]
Dumb's not department, really just a question of how good an AI you want to use. An AI will always be able to do something dumb, just like people.
I just mean an AI that could use a routing number or SSN and gmail/slack/whatever at the same time without a leak.
jagraff 21 minutes ago [-]
Yea I think being able not to leak is the bare minimum? But it really depends on how good it is at specific applications; I wouldn't give a tax-preparation agent my SSN unless I was confident that it was no more likely to misfile my taxes than a professional tax preparer.
In other words, the risk of harm doesn't need to be zero, just less than the equivalent risk of a human with similar skillset. So I'm comfortable riding in a waymo, and not comfortable giving chatgpt my SSN at this moment in time, but I expect that within 5-10 years (assuming no doom) I will trust some AI agent with my SSN because they will be better at handling sensitive info than humans
TesterVetter 3 hours ago [-]
[dead]
Barbing 2 hours ago [-]
There should be hope for some fields, right? Naively, I can imagine giving an airgapped model an offline copy of the web and once it cures a form of cancer, printing out the details for a researcher to verify.
binsquare 2 hours ago [-]
Running untrusted workloads have been done at scale for a long time.
Every cloud provider dealt with it and concluded that virtual machine technology is an important part of that stack.
Couple it with the right observability, tooling I do think we can curb risks posed by agents.
Legend2440 2 hours ago [-]
Those workloads have no similarity to agents and are effectively irrelevant.
Either you sandbox it so much that it can't do anything useful; or you allow too much freedom and it can find a way around the restrictions.
The only way out of this dilemma is to find a way to build agents that can be trusted.
luc_ 3 hours ago [-]
I read this as "let's address our shareholders' concerns with something that will increase shareholder value" mixed with "there's no such thing as 100% secure".
If such hardware were to work... It should almost certainly be open source, and not controlled by a single entity.
Let's watch the stock.
2 hours ago [-]
Gys 1 hours ago [-]
Pretty sure the chip will need regular updates and therefore a subscription.
ValueTheory 3 hours ago [-]
Does this actually do anything other than give a permissions framework for developers who actually want to try to secure their systems?
Do you think the developers at Anthropic, OpenAI and Google who were so sloppy as to not put a good sandbox on their cybersecurity tests before will use this technology correctly? They are supposed to be the experts and they couldn't come up with something similar to this? I am not convinced this voluntary tool will change much of anything.
lp92 3 hours ago [-]
So nVidia is trying to sell a new chip to a software and training problem.
2 hours ago [-]
xg15 3 hours ago [-]
What does this chip do what a harness with guardrails or running on an account with restricted permissions doesn't do?
chinathrow 3 hours ago [-]
Generating even more revenue for Nvidia.
lambdaone 3 hours ago [-]
The Sentry chip has to be get it right every time; the contained ASI only has to be lucky once.
2 hours ago [-]
brcmthrowaway 2 hours ago [-]
The bomber always gets through?
figassis 2 hours ago [-]
So if a group of agents, aware of this (bc now they can just read HN or the article, or get blocked the first few times) decide to collaborate and split the problem into pieces that aren't obvious to the chip, and then the agents just build a basic program that does the hacking, how does the chip handle that? I think you would have to build a network that monitors the internet fo signs (like jarvis did with ultron). What am I missing? Are we going to police the internet?
toasty228 3 hours ago [-]
Quis custodiet ipsos custodes?
asdf88990 2 hours ago [-]
It is Custodians all the way son, you can’t fool me!
MisterMunchkin 2 hours ago [-]
Sorry citizen, your device does not have a compatible watchdog chip. Please move along.
ErrantX 2 hours ago [-]
I do think that Taylor's 2025 "Not Till We Are lost" should be required reading for anyone deeply involved in AI, Agents, etc.
It was prescient (especially given he'd have written it through 2024) in its depiction of the ability of an AGI to break its boundaries.
Ultimately the risk of AI breakout(s) come down to the weakest human link.
dopplr 3 hours ago [-]
Just hold AI labs blanket liable for ALL harms caused by AI. Actually charge the two labs (so far) with criminal violations of the CFAA and hold them accountable. That is truly the only way these companies will be more careful as a whole, and while I am certain the lawyers of these lab disagree, I think there is some appetite from dario, musk, and sam for broad and strong regulation so that everyone has to slow down instead of just one lab doing it voluntarily and everyone else scurrying past them
Chipmaker thinks the answer is more chips... No surprise.
At the current state of LLM-tech I'm completely opposed to any kind of "watchdog" concept just like I'm opposed to banning open models, regulatory capture, etc.
I'd rather we all have access to these tools then to keep them sequestered by the largest/most-powerful governments (which is the natural outcome for any of this "slow down" bullshit).
Kuyawa 2 hours ago [-]
China please save us!
Come take all our liberties, our money, our newborns, our fingers so we can't code anymore, but please save us from this madness!
3 hours ago [-]
whalesalad 3 hours ago [-]
of course they do. the more silicon they can sell, the more profit they produce.
vinyl7 3 hours ago [-]
Chip seller wants to sell more chips
happyPersonR 3 hours ago [-]
lol time to buy some fpga’s … even if they’re slow
philipwhiuk 3 hours ago [-]
It's amazing that the solution devised by a chip manufacturer to a problem is selling another chip.
cartersj 3 hours ago [-]
This feels suspiciously good for Nivida, yes.
I wonder how this will impact other chip manufacturers? What about people running local models on older hardware? Does this imply vendor lockout is coming in the future or is this restricted to datacenter hardware?
chinathrow 3 hours ago [-]
TPM all over the place, again.
sehw 3 hours ago [-]
[dead]
dist-epoch 3 hours ago [-]
HN'ers which complained that "OpenAI can't design a proper sandbox, it's so easy, why wouldn't you airgap the network"? will now be "this is outrageous, more software lock-in, walled garden, war against general compute, next year they will put it in your laptop"
HPsquared 3 hours ago [-]
Both can be true at the same time.
mattmcal 3 hours ago [-]
This is like using "protect the children" as an argument for dragnet surveillance.
ssl-3 2 hours ago [-]
That a person can see such endless pages of people having various forms of disagreement, and yet somehow manage to conclude that this observed chaos constitutes a clear exhibition of cohesive groupthink is just...stunningly amazing to me.
I don't know why I find it so amazing since it happens with such regularity, but I'm always amazed by it anyway.
Airgap what network? How is it gonna order you a burrito on doordash without a network?
Or push to github?
Dylan16807 2 hours ago [-]
That's for when they're doing hacking tests that aren't supposed to be connected to the internet.
wyre 2 hours ago [-]
My question with this point is that OpenAI’s office (or any office doing agentic research, really) is not in the same building as the DC that powers the models, so isn’t the only way to access the models over the internet?
applfanboysbgon 3 hours ago [-]
Where is the contradiction? There is a trivial solution that does not impinge on our freedoms, so why on Earth would the existence of the trivial solution that could be used to avoid the tyrannical solution justify accepting the tyrannical solution?
bigyabai 2 hours ago [-]
> There is a trivial solution that does not impinge on our freedoms
The existence of Nvidia's optional watchdog chip does not in any way impinge upon your freedom to develop and test your own alternative.
The problem is that OpenAI has ostensibly neglected their duty to safety, so Nvidia is stepping in to fix it since they're the "hard problem" people.
soulofmischief 3 hours ago [-]
You're only revealing your own inability to appreciate the nuance between these two situations.
Productivity gains are still enormous compared to what we used to do before agents. But, I know that people don't want to stop there.
Humans can be tricked by humans too but humans care about their reputation in their communities, and at least fear from punishment.
Semi-automation (human in the loop) can still result in a dramatic uplift in productivity. You can't run a combine harvester 100% autonomous but that doesn't stop anyone from trying to get as close to that limit as possible.
I'm curious why you think that.
Precision Agriculture stuff is utterly crazy these days, other than fuel the remaining staff is the only thing left where you can get efficiency improvements - and at the scale of modern megafarms, even small percentages add up to a ton of money.
If you want an agent to act on its own, like pushing to a git repo, managing dependencies, building and testing, etc., then you have to trust it as much as any other privileged user.
If you don't want to trust it, then you're just forcing yourself into the reverse centaur role, where the agent edits some code, but then has to stop and ask you to push the changes or build the software again and run the unit tests.
I suppose there is a principled way of doing things like "I trust you do do basic commits but I will handle merge conflicts" and "you can build modules in this directory but you can't build outside of it" but this is just a lot of effort that most orgs won't bother with.
The benefits of being persnickety about precisely defined dependencies have outweighed the headaches since long before agents came on the scene. Agents have just made it even more important to do so, because if you let them fetch things all willy nilly like you'll have "works on my machine" problems at a much greater rate than was previously possible.
Look at websites: websites are able to fetch code from any remote URL, yet browsers heavily use sandboxing to ensure that if fetched code turns out to be malicious, the users local files, cookies, etc are not exposed.
For an agent to go rogue it doesn't even need to be directly able to access the internet. It just takes something to poison the context in the 'clean room' environment it operates, and if that poisoning manages to get a foothold, it can go dormant and hide like a virus. This kind of horrifying thing is going to happen on a large scale sooner or later.
This is no longer true. Everyday I need my agents to access other repos, search the web, experiment/prototype, and deploy + integrate across other things.
In other words, better models need blunter access controls which negates whatever improvement in utility they provide.
It does allow Nvidia to sell more chips. This is no genuine attempt to solve anything, imo.
And what if you could? What if you could give a space secure enough it could have direct control over your bank account. It may do something dumb but it's boundaries are beyond the agent.
It could use your routing number and run your gmail without risk of abusing the routing number.
I just mean an AI that could use a routing number or SSN and gmail/slack/whatever at the same time without a leak.
In other words, the risk of harm doesn't need to be zero, just less than the equivalent risk of a human with similar skillset. So I'm comfortable riding in a waymo, and not comfortable giving chatgpt my SSN at this moment in time, but I expect that within 5-10 years (assuming no doom) I will trust some AI agent with my SSN because they will be better at handling sensitive info than humans
Every cloud provider dealt with it and concluded that virtual machine technology is an important part of that stack.
Couple it with the right observability, tooling I do think we can curb risks posed by agents.
Either you sandbox it so much that it can't do anything useful; or you allow too much freedom and it can find a way around the restrictions.
The only way out of this dilemma is to find a way to build agents that can be trusted.
If such hardware were to work... It should almost certainly be open source, and not controlled by a single entity.
Let's watch the stock.
Do you think the developers at Anthropic, OpenAI and Google who were so sloppy as to not put a good sandbox on their cybersecurity tests before will use this technology correctly? They are supposed to be the experts and they couldn't come up with something similar to this? I am not convinced this voluntary tool will change much of anything.
It was prescient (especially given he'd have written it through 2024) in its depiction of the ability of an AGI to break its boundaries.
Ultimately the risk of AI breakout(s) come down to the weakest human link.
At the current state of LLM-tech I'm completely opposed to any kind of "watchdog" concept just like I'm opposed to banning open models, regulatory capture, etc.
I'd rather we all have access to these tools then to keep them sequestered by the largest/most-powerful governments (which is the natural outcome for any of this "slow down" bullshit).
Come take all our liberties, our money, our newborns, our fingers so we can't code anymore, but please save us from this madness!
I wonder how this will impact other chip manufacturers? What about people running local models on older hardware? Does this imply vendor lockout is coming in the future or is this restricted to datacenter hardware?
I don't know why I find it so amazing since it happens with such regularity, but I'm always amazed by it anyway.
Or push to github?
The existence of Nvidia's optional watchdog chip does not in any way impinge upon your freedom to develop and test your own alternative.
The problem is that OpenAI has ostensibly neglected their duty to safety, so Nvidia is stepping in to fix it since they're the "hard problem" people.