This reminds me of a friend whose company ran a daily perl script that committed every financial transaction of the day to a database. Without the script, the company could literally make no money irrespectively of sales because this database was one piece in a complex system for payment processor interoperability.
The script ran in a machine located at the corner of a cubicle and only one employee had the admin password. Nobody but a handful of people knew of the machine's existence, certainly not anyone in middle management and above. The script could only be updated by an admin.
Copilot may be good, but sure as hell doesn't know that admin password.
Everywhere I’ve ever worked has had that mission critical box.
At one of my jobs we had a server rack with UPS, etc, all the usual business. On the floor next to it was a dell desktop with a piece of paper on it that said “do not turn off”. It had our source control server in it, and the power button didn’t work. We did eventually move it to something more sensible but we had that for a long time
yeah. I mean, someone else would _eventually_ figure it out. There wasn't full disk encryption or anyhting on it, so if the guy got hit by a bus, and the machien turned off we probably would have just imaged the disk and got it running in a VM.
My last job at a telco I was in charge of a system that billed ~5 million dollars monthly.
When the machine was built, the guy that did it didn’t record the root password. He added me to sudoers before he left. I left a few years later, nobody took ownership.
Looking at the web interface, I can tell it’s still running, doing its thing. I’m sure its still running Linux from 2008.
That sounds pretty bad. Not a great argument against AI: "Our employees have created such a bad mess that AI wont work because only they know how the mess they created works".
> "Our employees have created such a bad mess that AI wont work because only they know how the mess they created works".
This is an ironclad argument against fully replacing employees with AI.
Every single organization on Earth requires the people who were part of creating the current mess to be involved in keeping the organization functioning.
Yes you can improve the current mess. But it's still just a slightly better mess and you still need some of the people around who have been part of creating the new mess.
Just run a thought experiment: every employee in a corporation mysteriously disappear from the face of the Earth. If you bring in an equal number of equally talented people the next day to run it, but with no experience with the current processes of the corporation, how long will it take to get to the same capability of the previous employees?
I think the parent is saying what if the AI made such a terrible mess that the team of imperfect people thought it was fine, but it was just as bad as the terrible mess the team would have created because the team is not capable of evaluating whether it's a good idea or not. (possible follow on consequences -- no one can debug it or figure out if it's a good idea either)
An old colleague and friend used to print out a 30 page perl script he wrote to do almost exactly this in this scenario. A stapled copy could always be found on his dining room table.
The script ran in a machine located at the corner of a cubicle and only one employee had the admin password. Nobody but a handful of people knew of the machine's existence, certainly not anyone in middle management and above. The script could only be updated by an admin.
Copilot may be good, but sure as hell doesn't know that admin password.