Path: utzoo!utgpu!news-server.csri.toronto.edu!cs.utexas.edu!rutgers!aramis.rutgers.edu!athos.rutgers.edu!nanotech From: dmocsny@minerva.che.uc.edu (Daniel Mocsny) Newsgroups: sci.nanotech Subject: Re: Some problems of super-intelligence Message-ID: Date: 12 Jan 91 03:36:40 GMT Sender: nanotech@athos.rutgers.edu Organization: University of Cincinnati, Cin'ti., OH Lines: 217 Approved: nanotech@aramis.rutgers.edu In article cognos!geovision!gd@dciem (Gord Deinstadt) writes: >dmocsny@minerva.che.uc.edu (Daniel Mocsny) writes: >>Think of the danger of biological viruses that exist today. Do you >>live in a hermetically sealed bubble? > >My brain does. Ever heard of the blood-brain barrier? Yes. However, your body has quite a number of failure modes that do not require its compromise. Your brain is reasonably sealed today, but nobody has yet out-lived their body. >>Defending against computer viruses is much easier than defending >>against biological viruses. Biological viruses have many avenues of >>entry into your body, and you are unable to close all of them. >>Computer viruses have only one avenue of entry: you download a >>set of instructions and run them. > >We are talking about a neural network here. In a neural network there >is no distinction between data and program. My mistake, then. I assumed we were talking about some sort of a high-fidelity neural-network simulation running on top of a relatively conventional computer (with adequate speed, etc.). The underlying computer will certainly (?) distinguish between program and data, if only to guard against the very possibility you fear. You could, for example, elect to buffer your neural network brain against fresh infusions of outside information by running them first in an isolated simulation. A neural network doesn't distinguish between data and program, but certainly the physical organization of our brains implies a distinction between network topology and sensory data. If this were not so, then perfect brainwashing would be possible. I.e., merely subject a person to the correct input data stream (sensory stimuli), and program their mental state to your liking. While people certainly can be persuaded of some things, limits seem to exist. For example, my argument is not likely to persuade you :-) And even if perfect brain-programming-via-sensory-stimuli is possible, consider what it implies: essentially drowning out all competing stimuli. Really effective brainwashing requires physically confining a victim and wearing down their resistance through physical stress. I expect your neural-network simulation would be similarly robust and defensive. For an invader to sneak in and corrupt your thoughts, it would have to mount an all-out attack. Since you would be starting with a substantial stock of running programs, redundant processes checking each other, and many connections to the outside, an invader might have to expend more resources than you have. Remember, if you start acting "too funny", your networked friends will notice. Essentially, they will be checking your behavior against their own gross models of your past behavior. This is, of course, pretty much how we detect the onset of mental disease today. Could someone corrupt you and all your friends at the same time? And all their friends? (Remember, some of your "friends" can be sub-processes running on your computer.) Possibly, but then we reach the fundamental realities of evolutionary competition. One brain, or a small group of brains, are most unlikely to be able to stay uncorrupted while they corrupt all the other brains. Recall the Fundamental Theorem of Conspiracy: "Every organization is infiltrated by clandestine agents from some competing organization. This competing organization is, recursively, infiltrated by clandestine agents from some other organization. In other words, 'Big spooks have little spooks upon their backs to bite them...'" The only thing that keeps us sane is this: no organization can commit 100% of its resources to infiltrating other organizations. For an organization to send out 1 clandestine agent, it must maintain some X supporting workers, where X>1. Thus, any infiltrating organization must itself have some organization susceptible to infiltration. Therefore we have competition. >It needn't be anything so obvious, anyway. Just edit the history books >and supply everyone with new "facts". If done gradually probably noone >would notice. Tyrants try to do this all the time, and to some extent >they succeed even though they don't have direct access to the brain. Hold on! Tyrants succeed *precisely* because they are exploiting *information poverty*. If the cost of processing information is very high, then the power to do so accumulates into a few hands. However, as the cost of processing information drops, the power to process information *necessarily* distributes. When books cost a fortune to own or create, then only the privileged few can own or create them. Not surprisingly, the privileged few exploit their information power to maintain their privilege. But make books so cheap that everybody can own and create them, and the privileged few can no longer hope to control what will be in those books. Remember, democracy resulted from a technological innovation that reduced the cost of information power: the printing press. >Even sincere people do it, convincing other people and themselves that >it didn't really happen that way... if your memories are in someone >else's care, you are no longer free. But I don't see why your memories should be in someone else's care. You can make all the backups you want, stick in all the CRC you like, maintain a whole sequence of progressively more approximate models which check each other, etc. You aren't going to leave yourself wide open. You'll have an immune system. Your body doesn't leave itself open. It has multiple lines of defense against invaders. A big gob of protoplasm like you or me looks like a hell of a food source to all the microbes out there. But the dinner is quite capable of striking back. What if your immune system gets corrupted? Well, it does happen (e.g., AIDS). However, this gets back to the Fundamental Theorem of Conspiracy. Not *everybody* can simultaneously be totally corrupted. Well, maybe. :-) >> Also, when a biological virus >>appears, there is nobody we can potentially locate and throw in jail. > >Which might deter the small fry, but the danger is the big fish, >like politician X. I believe you are still thinking against the historical backdrop of *information poverty*. When people can't process enough information to support their basic needs, then division of labor is necessary, and power accumulates into a few hands. The rise of information power will tear down the social structures that today grant politician X the power to abuse. Just watch. Heck, just look at Eastern Europe. You don't even need much information power at all. When everybody has tera-MIPS or whatever, how in the world is politician X going to get away with anything? What's he going to have, tera-tera-MIPS? >>Therefore, you will allocate some of your computer resources to a >>logical immune system, just as you today allocate some of your >>biological resources to a biological immune system. > >Can you suggest a mechanism, in principle, for such a thing? Yes. 1. You maintain a series of approximate representations of your brain state. Each of these connects with the outside world via a restricted interface. Your innermost "self" obtains outside information only after a time delay, and only through the indirect agency of your approximate selves. Each layer constantly monitors neighboring layers for damage and/or suspicious behavior, and doesn't pass anything inward until it has satisfied itself that everything is fine. The royalty of old had "royal tasters". To guard against being poisoned by conspirators, the royalty had someone else taste all their food first. Similarly, an advancing army always sends out scouts and patrols. No doubt a lot of scouts get killed. But this is the price the army pays to learn about threats. 2. You maintain multiple outside connections which compete with each other. Today, if you start to go insane, your friends will notice and become concerned. Your friends are unlikely to go insane at exactly the same time that you do. >The only thing I can think of is to somehow label data as coming >from "out there". Perhaps the only safe connection is via the senses, >ie. you hear the computer's voice. But this severely limits what >it can do for you. Safety always limits. However, death is considerably more limiting. You can't compare your practical capability with some ideal capability that would exist if you had no outside threats. You always have outside threats. The cost of those threats is not your money, any more than the taxes Uncle Sam takes out of your check is your money. Think of how much cheaper a house could be if we could be sure the weather would never get ugly. Well, the weather *does* get ugly. >By the same token I might well accept an ROM library implant in >my head; it's the network connection I would reject. I want to >control when that library changes. How do you know the ROM wouldn't have "back doors"? :-) In any case, even with a network connection, "you" shouldn't have any trouble controlling when your library changes. Every interface between two systems is restricted in some way (or else they wouldn't be two systems). >>computer code is so much easier to analyze and work with (compared >>to protoplasm) that I think a great advantage accrues to the >>defender. > >Ah, but so far the pathogens our bodies have had to deal with have >been created by blind evolution, not by intelligent and hostile >entities. However, that blind evolution has quite a head start. Despite all our hostility and intelligence, we are not yet capable of constructing, from simple chemical reagents, a biological invader as effective as the AIDS virus. Remember, most conflicts between humans are peer-to-peer conflicts. In virtually every sustained war, neither side maintains a unilateral, overpowering technological edge. Whenever one side invents the "secret weapon" that will grant them supremacy, the other side gets it shortly thereafter. That is because engineering is harder than reverse engineering. The spread of information power tends to level the field, not create more pockets of concentrated threat. -- Dan Mocsny Snail: Internet: dmocsny@minerva.che.uc.edu Dept. of Chemical Engng. M.L. 171 dmocsny@uceng.uc.edu University of Cincinnati 513/751-6824 (home) 513/556-2007 (lab) Cincinnati, Ohio 45221-0171