1 Regulatory lag

“As SEC Listens to HFT and Exchanges, Europe Drives Discussion”:

CC Licenced

Regulatory lags for New Technology [2013 notes]

5 gwern 31 May 2017 01:27AM

I found some old notes from June 2013 on time delays in how fast one can expect Western political systems & legislators to respond to new technical developments.

In general, response is slow and on the order of political cycles; one implication I take away is that a takeoff an AI could happen over half a decade or more without any meaningful political control and would effectively be a ‘fast takeoff’, especially if it avoids any obvious mistakes.

1 Regulatory lag

“Regulatory delay” is the delay between the specific action required by regulators or legislatures to permit some new technology or method and the feasibility of the technology or method; “regulatory lag” is the converse, then, and is the gap between feasibility and reactive regulation of new technology. Computer software (and artificial intelligence in particular) is mostly unregulated, so it is subject to lag rather than delay.

Unfortunately almost all research seems to focus on modeling lags in the context of heavily regulated industries (especially natural monopolies like insurance or utilities), and few focus on compiling data on how long a lag can be expected between a new innovation or technology and its regulation. As one would expect, the few results point to lags on the order of years; for example, Ippolito 1979 (“The Effects of Price Regulation in the Automobile Insurance Industry”) finds that the period of price changes goes from 11 months in unregulated US states to 21 months in regulated states, suggesting the price-change framework itself causes a lag of almost a year.

Below, I cover some specific examples, attempting to estimate the lags myself:

HFT: >5 years
Self-driving cars: >6 years
Hacking: ~30 years

(Nuclear weapons would be an interesting example but it’s hard to say what ‘lag’ would be inasmuch as they were born in government control and are subject to no meaningful global control; however, if the early proposals for a world government or unified nuclear weapon organization had gone through, they would also have represented a lag of at least 5 years.)

1.1 Hacking

Computer hacking existed for quite a while before relevant laws were passed and serious law enforcement began, which is typically considered to have begun with Operation Sundevil:

Prior to 1990, people who manipulated telecommunication systems, known as phreakers, were generally not prosecuted within the United States. The majority of phreakers used software to obtain calling card numbers and built simple tone devices in order to make free telephone calls. A small elite, and highly technical segment of phreakers were more interested in information about the inner workings of the telecommunication system than in making free phone calls. Phone companies complained of financial losses from phreaking activities.[5] The switch from analog to digital equipment began to expose more of the inner workings of telephone companies as hackers began to explore the inner workings, switches and trunks. Due to a lack of laws and expertise on the part of American law enforcement, few cases against hackers were prosecuted until Operation Sundevil.[4]

However, starting in 1989, the US Secret Service (USS), which had been given authority from Congress to deal with access device fraud as an extension of wire fraud investigations under Title 18 (§ 1029), began to investigate. Over the course of the 18 month long investigation, the USS gathered alleged evidence of rampant credit card and calling card fraud over state lines.[6]

This gives a time-delay of decades from the first phreaks (eg. Steve Jobs & Wozniak selling blue boxes in 1971 after blue boxing was discovered in the mid-60s) to the mid-1980s with the passage of the Computer Fraud and Abuse Act & Computer Security Act of 1987; prosecution was sporadic and light even after that, for example Julian Assange as Mendax in 1991 was raided and ultimately released with a fine in 1995. (Since then, at least 49 states have passed laws dealing with hacking with an international convention spreading post-2001.)

1.2 High frequency trading

HFT, while apparently only becoming possible in 1998, was marginal up until 2005 where it grew dramatically and became controversial with the 2010 flash crash and the 2012 Knight Capital fiasco. Early SEC rule-changes did little to address the issue; no US legislation has been passed, or appears viable given Wall Street lobbying. European Parliament legislation is pending, but highly controversial with heavy lobbying from London. Otherwise, legislation has been passed only in places that are irrelevant (eg. Germany). Given resistance in NYC & London, and the slow movement of the SEC, there will not be significant HFT regulation (for better or worse) for years to come, and it is likely irrelevant as the area matures and excess profits disappear - “How the Robots Lost: High-Frequency Trading’s Rise and Fall”.

“Insight: Chicago Fed warned on high-frequency trading”, Reuters:

More than two years ago, the Federal Reserve Bank of Chicago was pushing the Securities and Exchange Commission to get serious about the dangers of super-fast computer-driven trading. Only now is the SEC getting around to taking a closer look at some of those issues…Even as the SEC gears up for a meeting on Tuesday to discuss software glitches and how to tame rapid-fire trading, the eighth public forum it has had in two years on market structure issues, regulators in Canada, Australia and Germany are moving ahead with plans to introduce speed limits to safeguard markets from the machines…To be sure, it is not as if the SEC has simply stood idly by and allowed the machines to run amok. The agency did put in place some new safeguards such as circuit breakers on stocks, after the May 2010 flash crash. The circuit breakers are intended to prevent a market-wide crash by briefly halting trading in particular stocks displaying sharp price moves within a 5-minute window, giving the algorithms a chance to let go of trading patterns that may have turned into vicious cycles…And recently, the SEC fined the New York Stock Exchange’s operator, NYSE Euronext, $5 million for allegedly giving some customers “an improper head start” on proprietary trading information.

“High speed trading begets high speed regulation: SEC response to flash crash, rash”, Serritella 2010:

The SEC has been quick to react to the Flash Crash, determined to avoid such market disruptions in the future.57 On June 10, 2010, the SEC published new rules (Rules), which require trading centers to halt trading in certain individual securities and derivatives if pricing thresholds are reached.58 Until December 10, 2010, the Rules are in a pilot period so that they may be adjusted and expanded.59

…The SEC published its new Rules slightly over a month after the Flash Crash and did so in an expedited manner in order ―to prevent a recurrence‖ of the May 6, 2010 market disruptions. RULES, supra note 5, at 4. ―The Commission believes that accelerating approval of these proposals is appropriate as it will enable the Exchanges nearly immediately to begin coordinating trading pauses across markets in the event of sudden changes in the value of the S&P 500 Index stocks.‖ Id. at 12. The Commission was ―concerned that events such as those that occurred on May 6 can seriously undermine the integrity of the U.S. securities markets. Accordingly, it is working on a variety of fronts to assess the causes and contributing factors of the May 6 market disruption and to fashion policy responses that will help prevent a recurrence.‖ Id. at 4.

…The new Rules tighten the thresholds and, for the first time, centralize the control of circuit breakers.63 Circuit breakers simply refer to the ability of exchanges to temporarily halt trading in a security or derivative to avert sell-offs during periods of extreme downward pressure, or to close the markets before the end of the normal trading day.64 While, previously, the exchanges each controlled their own circuit breakers, they all generally adhered to the thresholds and formulas set forth in NYSE Rule 80B.65 Rule 80B has three different thresholds-10%, 20% and 30%-each of which is tied to the DJIA and if met would result in a ―time out‖ to market activity altogether on any exchange to execute a circuit breaker mechanism.66 Despite the extreme price movements of May 6, 2010, the circuit breakers’ lowest threshold was not met, as, at its worst point in the Flash Crash, the DJIA was down 9.16%-lower than the 10% drop required to trigger the circuit breakers under Rule 80B.67 The SEC, recognizing that using the DJIA as a benchmark for circuit breakers may obscure extreme price movements in individual securities or derivatives, has extended the utility of circuit breakers to target individual securities and derivatives.68 Under the new Rules, the exchanges are required to issue five minute trading halts in a security if the price of that security moves at least 10% in either direction from its price in the preceding five minute period.69 To avoid interfering with the openings and closings of markets, these requirements are only in force from 9:45 a.m. to 3:35 p.m.70 The Rules do not displace Rule 80B’s mandates, rather they supplement their preexisting coverage with the ability to target individual securities whose volatility may not have enough of an effect on the DJIA to otherwise trigger a circuit breaker.71

…While still in their infancy, the new Rules suffer from crucial limitations which threaten their efficacy, not the least of which is that they only apply to stocks in the S&P 50075 and Russell 100076 indexes as well as select derivatives.77…For example, since the SEC’s new circuit breaker requirements do not only apply to all securities and derivatives, it is possible that trading could be halted in a given security while sales in one of its derivatives continue unabated, thus, frustrating the exchanges’ congressional mandate to promote market integrity and protect investors79 as well as fostering a disconnect between the prices of the derivative-an ETF, for example-and its underlying trading-halted security.

“U.S. Leads in High-Frequency Trading, Trails in Rules”, Bloomberg editors op-ed:

In contrast to this go-slow approach, Germany, Canada, Australia and the European Union are taking up some of the tools the U.S. should consider to keep computerized trading from running amok. To cite a few examples:

In Germany, legislation is pending to require high- frequency traders to register so regulators can better track their market moves.

Canada charges fees to firms that attempt to clog markets with buy and sell orders, as well as cancellations, a practice known as quote stuffing. High-frequency trading firms sometimes do this to overload the less-sophisticated trading systems of rivals and exploit minuscule and fleeting price discrepancies.

Australia will ask trading firms to conduct stress tests to gauge how they deal with market shocks.

The EU is reviewing a number of measures including one that would require a trading firm to honor a bid for half a second, a lifetime in a market where trades can be executed in microseconds.

It wouldn’t hurt if U.S. regulators took a look at these options and considered a few others, as well.

“SEC Leads From Behind as High-Frequency Trading Shows Data Gap”, Businessweek:

The U.S. Securities and Exchange Commission, stung by criticism that it lacks the knowledge to analyze the computerized trading that has come to dominate American stock markets, is planning to catch up. Initiatives to increase the breadth of data received from exchanges and to record orders from origination to execution are at the center of the effort. Gregg Berman, who holds a doctorate in physics from Princeton University, will head the commission’s planned office of analytics and research…“It’s amazing it’s taken 30 years,” Weild, a former vice chairman of NASDAQ Stock Market, said in a phone interview. “Meanwhile, there’s been an arms race on Wall Street and the SEC is outclassed in its ability to reconstruct events and look for vulnerabilities.”…The audit trail won’t be in place for several years and the industry hasn’t figured out how much it will cost and who will pay for it. Midas [overnight analysis] will be fully rolled out by the end of 2012, Berman said. It won’t include information about the one-third of trading that occurs away from exchanges.

In the City of London and European trading centers, the wait is on for the publication this week of the regulatory proposals previewed last week by the European Parliament’s Economic and Monetary Affairs Committee, which sent shockwaves through the worldwide HFT community. The Committee proposed a mandated half-second freeze for all trading orders, not only in equities, but every market, including fixed income and other asset classes. While the requirement to keep all trades alive for at least 500 milliseconds went far beyond what was expected in terms of orders, markets analyst Rebecca Healey told MNI there is believed to be an even more game-changing proposal in the document itself, one that could threaten the use of so-called dark pools by investors.

“City of London opposes tighter regulation of high-frequency trading”, Financial News:

MEPs [Members of European Parliament] this week unanimously voted through rules that will severely limit the controversial share-trading practice of high frequency trading, as part of a financial sector reform bill…Members of the economic and monetary affairs committee voted through proposals under the revised Markets in Financial Derivatives legislation - known as Mifid 2…Mifid 2 also includes measures designed to limit speculation on commodity markets, which has been blamed for distorting food prices and harming the world’s poorest populations. And there are rules aimed at protecting investors from being sold inappropriate products. Last week the Bureau reported that MEPs were planning tough new curbs on HFT, despite stiff opposition from the City of London and the wider financial sector. Stock exchanges, which receive a significant portion of their income from trading fees and high-tech services for HFT, were said to be ‘vocal’ about their desire to avoid regulation. UK-based stock exchanges have argued that they already have circuit breakers so there is no need to officially mandate them…But it would be premature to announce HFT’s demise in Europe: the bill still has a way to go before it becomes law. The three-tiered structure of European lawmaking means that the legislation was previously passed by the European Commission and draft legislation will now go before the European Union’s finance ministers. The three versions will then be reconciled through a ‘trialogue’ process.

The move to put the brakes on HFT across Europe has met stiff opposition from the City of London and the UK government.‘We must be careful not to introduce measures based on the assumption that high frequency trading is, per se, harmful to markets,’ warned the Financial Services Authority (FSA), responding to the draft legislation on behalf of the government. It rejected several of the proposed measures. The British Bankers’ Association described the requirement for some traders to become market makers as ‘particularly onerous’. The Treasury supports HFT arguing it brings liquidity to markets and reduces costs. Exchanges and the HFT industry have campaigned against the measures in public and behind closed doors. Exchanges have been ‘very vocal’ about HFT, and a key priority has been to ‘preserve all HFT’, Kay Swinburne told the Bureau…Since January 2010, the LSE and its lobbyists, City law firm Freshfields, have met MEPs from the three main British parties at least 15 times to discuss Mifid and similar legislation, lobbying registers show. Freshfields’ public affairs director Christiaan Smits met with the Conservatives’ lead on Mifid, Dr Kay Swinburne, eight times in two years on behalf of the LSE. Other intense discussions have been going on behind the scenes. In total, exchanges including Nasdaq, Deutsche Borse, NYSE Euronext, Bats Europe and Chi-X, and public affairs firms hired by them, have met with British MEPs at least 49 times over the period to discuss Mifid 2. Fleishman Hillard earned up to €50,000 (£40,000) each last year from representing exchange Chi-X and trading platform Equiduct, as well as up to €150,000 representing investment company Citadel, which has a substantial HFT arm. Specialist HFT companies have banded together to form a campaign group, the FIA European Principal Traders’ Association (Epta), which has issued position papers and lobbied politicians in the UK and EU…Epta has paid Brussels-based lobbyist Hume Brophy at least €100,000 last year to make its case in Brussels, EU lobbying registers show.

“Yet again, the UK government has sided with the robotraders on a Robin Hood Tax”, New Statesman:

Yet as the Bureau for Investigative Journalism revealed last week, of a 31-member panel tasked by the UK Government to assess Mifid II, 22 members were from the financial services, 16 linked to the HFT industry. A study by the Bureau last year revealed that over half the funding for the Conservative Party came from the financial sector, 27 per cent coming from hedge funds, financiers and private equity firms. This perhaps helps explain how the interests of a select group of traders get confused with the interests of the economy as a whole…Yet the UK Government has again chosen to stand apart in blocking a Europe wide-FTT, turning down billions in desperately needed revenue that could help save jobs, protect the poorest and avoid the worst in cuts to public services. Instead, advice of previous Party Treasurers Michael Spencer and Peter Cruddas was heeded, who infamously lobbied against the FTT. Both incidentally own multi-million pound financial firms which would be hit by such a tax.

“French Fin Min: Need regulation of high frequency trading”, FrenchTribune.com:

The Finance Minister of France, Christine Lagarde said on Thursday that there is a requirement for more regulation of high frequency trading with the majority of the effects seeming to be negative and resulting in artificial moves in the markets…However, she is having conflicts with UK’s Financial Services Authority regarding the regulation of high frequency trading firms, which make use of automated software and super-fast telecommunications networks which is capable of trading in milliseconds.

“German government to propose tighter regulation of high-frequency trading”, Washington Post:

Germany’s Finance Ministry said Tuesday a draft law will be considered by Chancellor Angela Merkel’s Cabinet on Wednesday. The bill would require traders to get special permission before they can deploy computers to carry out millions of trades a second to exploit split-penny price differences. Such trades would also have to be specially labeled and stock exchanges would need to ensure trading can quickly be suspended when an error occurs.

“Super funds want computer trading checks”

The $46 billion AustralianSuper said HFT could make a “positive contribution” to financial markets but some strategies were designed to exploit other participants and harmed market integrity. “HFT strategies that are manipulative in nature . . . are problematic and ultimately raise the cost of investing and unfairly redistribute profits,” said Innes McKeand, AustralianSuper head of equities…Associations such as Industry Super Network called for a crackdown on HFT before it became dominant in Australia. The Australian Council of Trade Unions also asked for a ban on HFT until regulators “completed a detailed assessment” of the role of such trades. Russell Investments head of implementation in Australia Adam van Ness said HFT volumes were increasing locally. However, there was no straight answer as to whether HFT provided perks, or hindered, investors and traders. “It’s an open debate of whether it’s good or bad,” said Ness. The Australian Securities and Investments Commission “is taking a closer look at this - the outcome of which will make [HFT] a bit more restrictive here”…“In particular, a kill-switch requirement might have limited the extent of the Knight Capital losses as it could have facilitated a speedier termination of faulty orders.”

“High frequency trading and its impact on market quality”, Brogaard 2010:

Congress and regulators have begun to take notice and vocalize concern with HFT. The Securities and Exchange Commission (SEC) issued a Concept Release regarding the topic on January 14, 2010 requesting feedback on how HFTs operate and what benefits and costs they bring with them (SEC, January 14, 2010). The Dodd Frank Wall Street Reform and Consumer Protection Act calls for an in depth study on HFT (Section 967(2)(D)). The Commodity Futures Trading Commission (CFTC) has created a technology advisory committee to address the development of high frequency trading. Talk of regulation on HFT has already begun. Given the lack of empirical foundation for such regulation, the framework for regulation is best summarized by Senator Ted Kaufman, “Whenever you have a lot of money, a lot of change, and no regulation, bad things happen” (Kardos and Patterson, January 18, 2010). There has been a proposal (House Resolution 1068) to impose a per-trade tax of .25%.

“The rise of computerized high frequency trading: use and controversy”, McGowan 2010:

The dramatic increase in HFT is most likely due to its profitability. Despite the economic recession, high- frequency trading has been considered by many to be the biggest “cash cow” on Wall Street and it is estimated that it generates approximately $15- $25 billion in revenue.16 [See Tyler Durden, “Goldman’s $4 Billion High Frequency Trading Wildcard”, ZEROHEDGE (Jul. 17, 2009, 2:16AM) (discussing estimates by the FIX Protocol, an organization that maintains a messaging standard for the real-time electronic exchange of securities transactions).]

Office space in such areas sometimes costs an astronomical amount, but firms are willing to pay for it. For instance, in Chicago, 6 square feet of space in the data center where the exchanges also house their computers can go for $2,000 or more a month.57 Despite these high prices, the number of firms that co-locate at exchanges such as the NASDAQ has doubled over the last year.58 57 See Moyer & Lambert, supra note 6 (stating that some trading firms even spend 100 times that much to house their servers). 58 Sal L. Arnuk & Joseph Saluzzi, “Toxic Equity Trading Order Flow on Wall Street: The Real Force Behind the Explosion in Volume and Volatility”, THEMIS TRADING LLC WHITE PAPER.

Today, latency arbitragers use algorithms to create models of great complexity that can involve hundreds of securities in many different markets. This practice is highly lucrative. For instance, the financial markets research and advisory firm TABB Group has estimated that annual aggregate profits of low-latency arbitrage strategies exceed $21 billion, an amount which is spread out among the few hundred firms that deploy them. [See Iati, supra note 13 (quoting TABB group’s estimate).]… Because of high frequency trading’s prominence, the next few years of changing regulations will be extremely interesting. The SEC has a very difficult job ahead of it in attempting to regulate these innovative practices while at the same time upholding the agency’s primary concerns: protecting the average investor and ensuring markets remain relatively efficient.

…Additionally, the lack of regulation on naked access allows a reckless high frequency trader to conceivably pump out hundreds of thousands of faulty orders in the two minute period it typically takes to rectify a trading system glitch. [See MOYER & LAMBERT, supra note 6.] Sang Lee, a market analyst from Aite Group, believes that “[i]n the worst case scenario, electronic fat fingering or intentional trading fraud could take down not only the sponsored participants, but also the sponsoring broker and its counterparties, leading to an uncontrollable domino effect that would threaten overall systematic market stability.”107 Because of these doomsday scenarios and others advanced by some Democratic lawmakers, the SEC will most likely propose rules to limit this practice in the upcoming months.

“High-Speed Trading No Longer Hurtling Forward”, 14 October 2012:

Profits from high-speed trading in American stocks are on track to be, at most, $1.25 billion this year, down 35% from last year and 74% lower than the peak of about $4.9 billion in 2009, according to estimates from the brokerage firm Rosenblatt Securities. By comparison, Wells Fargo and JPMorgan Chase each earned more in the last quarter than the high-speed trading industry will earn this year. While no official data is kept on employment at the high-speed firms, interviews with more than a dozen industry participants suggest that firms large and small have been cutting staff, and in some cases have shut down. The firms also are accounting for a declining percentage of a shrinking pool of stock trading, from 61% three years ago to 51% now, according to the Tabb Group, a data firm…The challenges facing speed-focused firms are many, the biggest being the drop in trading volume on stock markets around the world in each of the last four years. This has made it harder to make profits for traders who quickly buy and sell shares offered by slower investors. In addition, traditional investors like mutual funds have adopted the high-speed industry’s automated strategies and moved some of their business away from the exchanges that are popular with high-speed traders. Meanwhile, the technological costs of shaving further milliseconds off trade times has become a bigger drain on many companies…At the same time that the firms are making trims, regulators around the world have increased their scrutiny of high-speed traders, and the structure of the financial markets has continued to shift. Executives at the trading firms worry that new regulations could curtail business even more, but so far regulators in the United States have taken few steps to rein in trading practices…The contraction is also pushing the firms to move into trading of other financial assets, like international stocks and currencies. High-speed firms accounted for about 12% of all currency trading in 2010; this year, it is set to be up to 28 percent, according to the consulting firm Celent. But executives at several high-speed firms said that trading in currencies and other assets was not making up for the big declines in their traditional areas of United States stocks, futures and options. Sun Trading in Chicago bought a firm that allowed it to begin the automated trading of bonds earlier this year. That did not make up for the 40 employees the company cut in 2011.

1.3 Self-driving cars

LW post

The first success inaugurating the modern era can be considered the 2005 DARPA Grand Challenge where multiple vehicles completed the course. The first legislation of any kind addressing autonomous cars was Nevada’s 2011 approval. 5 states have passed legislation dealing with autonomous cars.

However, these laws are highly preliminary and all the analyses I can find agree that they punt on the real legal issues of liability; they permit relatively little.

1.3.1 Lobbying, Liability, and Insurance

(Warning: legal analysis quoted at length in some excerpts.)

“Toward Robotic Cars”, Thrun 2010 (pre-Google):

Junior’s behavior is governed by a finite state machine, which provides for the possibility that common traffic rules may leave a robot without a legal option as how to proceed. When that happens, the robot will eventually invoke its general-purpose path planner to find a solution, regardless of traffic rules. [Raising serious issues of liability related to potentially making people worse-off]

“Google Cars Drive Themselves, in Traffic” (PDF), NYT 2010:

But the advent of autonomous vehicles poses thorny legal issues, the Google researchers acknowledged. Under current law, a human must be in control of a car at all times, but what does that mean if the human is not really paying attention as the car crosses through, say, a school zone, figuring that the robot is driving more safely than he would? And in the event of an accident, who would be liable - the person behind the wheel or the maker of the software?

“The technology is ahead of the law in many areas,” said Bernard Lu, senior staff counsel for the California Department of Motor Vehicles. “If you look at the vehicle code, there are dozens of laws pertaining to the driver of a vehicle, and they all presume to have a human being operating the vehicle.” The Google researchers said they had carefully examined California’s motor vehicle regulations and determined that because a human driver can override any error, the experimental cars are legal. Mr. Lu agreed.

“Calif. Greenlights Self-Driving Cars, But Legal Kinks Linger”:

For instance, if a self-driving car runs a red light and gets caught, who gets the ticket? “I don’t know - whoever owns the car, I would think. But we will work that out,” Gov. Brown said at the signing event for California’s bill to legalize and regulate the robotic cars. “That will be the easiest thing to work out.” Google co-founder Sergey Brin, who was also at the ceremony, jokingly said “self-driving cars don’t run red lights.” That may be true, but Bryant Walker Smith, who teaches a class at Stanford Law School this fall on the law supporting self-driving cars, says eventually one of these vehicles will get into an accident. When it does, he says, it’s not clear who will pay.

…Or is it the company that wrote the software? Or the automaker that built the car? When it came to assigning responsibility, California decided that a self-driving car would always have a human operator. Even if that operator wasn’t actually in the car, that person would be legally responsible. It sounds straightforward, but it’s not. Let’s say the operator of a self-driving car is inebriated; he or she is still legally the operator, but the car is driving itself. “That was a decision that department made - that the operator would be subject to the laws, including laws against driving while intoxicated, even if the operator wasn’t there,” Walker Smith says…Still, issues surrounding liability and who is ultimately responsible when robots take the wheel are likely to remain contentious. Already trial lawyers, insurers, automakers and software engineers are queuing up to lobby rule-makers in California’s capital.

“Google’s Driverless Car Draws Political Power: Internet Giant Hones Its Lobbying Skills in State Capitols; Giving Test Drives to Lawmakers”, WSJ, 12 October 2012:

Overall, Google spent nearly $9 million in the first half of 2012 lobbying in Washington for a wide variety of issues, including speaking to U.S. Department of Transportation officials and lawmakers about autonomous vehicle technology, according to federal records, nearing the $9.68 million it spent on lobbying in all of 2011. It is unclear how much Google has spent in total on lobbying state officials; the company doesn’t disclose such data.

…In most states, autonomous vehicles are neither prohibited nor permitted-a key reason why Google’s fleet of autonomous cars secretly drove more than 100,000 miles on the road before the company announced the initiative in fall 2010. Last month, Mr. Brin said he expects self-driving cars to be publicly available within five years.

In January 2011, Mr. Goldwater approached Ms. Dondero Loop and the Nevada assembly transportation committee about proposing a bill to direct the state’s department of motor vehicles to draft regulations around the self-driving vehicles. “We’re not saying, ‘Put this on the road,’” he said he told the lawmakers. “We’re saying, ‘This is legitimate technology,’ and we’re letting the DMV test it and certify it.” Following the Nevada bill’s passage, legislators from other states began showing interest in similar legislation. So Google repeated its original recipe and added an extra ingredient: giving lawmakers the chance to ride in one of its about a dozen self-driving cars…In California, an autonomous-vehicle bill became law last month despite opposition from the Alliance of Automobile Manufacturers, which includes 12 top auto makers such as GM, BMW and Toyota. The group had approved of the Florida bill. Dan Gage, a spokesman for the group, said the California legislation would allow companies and individuals to modify existing vehicles with self-driving technology that could be faulty, and that auto makers wouldn’t be legally protected from resulting lawsuits. “They’re not all Google, and they could convert our vehicles in a manner not intended,” Mr. Gage said. But Google helped push the bill through after spending about $140,000 over the past year to lobby legislators and California agencies, according to public records

As with California’s recently enacted law, Cheh’s [Washington D.C.] bill requires that a licensed driver be present in the driver’s seat of these vehicles. While seemingly inconsequential, this effectively outlaws one of the more promising functions of autonomous vehicle technology: allowing disabled people to enjoy the personal mobility that most people take for granted. Google highlighted this benefit when one of its driverless cars drove a legally blind man to a Taco Bell. Bizarrely, Cheh’s bill also requires that autonomous vehicles operate only on alternative fuels. While the Google Self-Driving Car may manifest itself as an eco-conscious Prius, self-driving vehicle technology has nothing to do with hybrids, plug-in electrics or vehicles fueled with natural gas. The technology does not depend on vehicle make or model, but Cheh is seeking to mandate as much. That could delay the technology’s widespread adoption for no good reason…Another flaw in Cheh’s bill is that it would impose a special tax on drivers of autonomous vehicles. Instead of paying fuel taxes, “Owners of autonomous vehicles shall pay a vehicle-miles travelled (VMT) fee of 1.875 cents per mile.” Administrative details aside, a VMT tax would require drivers to install a recording device to be periodically audited by the government. There may be good reasons to replace fuel taxes with VMT fees, but greatly restricting the use of a potentially revolutionary new technology by singling it out for a new tax system would be a mistake.

“Driverless cars are on the way. Here’s how not to regulate them.”

“How autonomous vehicle policy in California and Nevada addresses technological and non-technological liabilities”, Pinto 2012:

The State of Nevada has adopted one policy approach to dealing with these technical and policy issues. At the urging of Google, a new Nevada law directs the Nevada Department of Motor Vehicles (NDMV) to issue regulations for the testing and possible licensing of autonomous vehicles and for licensing the owners/drivers of these vehicles. There is also a similar law being proposed in California with details not covered by Nevada AB 511. This paper evaluates the strengths and weaknesses of the Nevada and California approaches

Another problem posed by the non-computer world is that human drivers frequently bend the rules by rolling through stop signs and driving above speed limits. How does a polite and law-abiding robot vehicle act in these situations? To solve this problem, the Google Car can be programmed for different driving personalities, mirroring the current conditions. On one end, it would be cautious, being more likely to yield to another car and strictly following the laws on the road. At the other end of the spectrum, the robocar would be aggressive, where it is more likely to go first at the stop sign. When going through a four-way intersection, for example, it yields to other vehicles based on road rules; but if other cars don’t reciprocate, it advances a bit to show to the other drivers its intention.

However, there is a time period between a problem being diagnosed and the car being fixed. In theory, one would disable the vehicle remotely and only start it back up when the problem is fixed. However in reality, this would be extremely disruptive to a person’s life as they would have to tow their vehicle to the nearest mechanic or autonomous vehicle equivalent to solve the issue. Google has not developed the technology to approach this problem, instead relying on the human driver to take control of the vehicle if there is ever a problem in their test vehicles.

[previous Lu quote about human-centric laws] …this can create particularly tricky situations such as deciding whether the police should have the right to pull over autonomous vehicles, a question yet to be answered. Even the chief counsel of the National Highway Traffic Safety Administration admits that the federal government does not have enough information to determine how to regulate driverless technologies. This can become a particularly thorny issue when there is the first accident between autonomous and self driving vehicles and how to go about assigning liability.

This question of liability arose during an [unpublished 11 Feb 2012] interview on the future of autonomous vehicles with Roger Noll. Although Professor Noll hasn’t read the current literature on this issue, he voiced concern over what the verdict of the first trial between an accident between an autonomous vehicle and normal car will be. He believes that the jury will almost certainly side with the human driver despite the details of the case, as he eloquently put in his husky Utah accent and subsequent laughter, “how are we going to defend the autonomous vehicle; can we ask it to testify for itself?” To answer Roger Noll’s question, Brad Templeton’s blog elaborates how he believes that liability reasons are a largely unimportant question for two reasons. First, in new technology, there is no question that any lawsuit over any incident involving the cars will include the vendor as the defendant so potential vendors must plan for liability. For the second reason, Brad Templeton makes an economic argument that the cost of accidents is borne by car buyers through higher insurance premiums. If the accidents are deemed the fault of the vehicle maker, this cost goes into the price of the car, and is paid for by the vehicle maker’s insurance or self- insurance. Instead, Brad Templeton believes that the big question is whether the liability assigned in any lawsuit will be significantly greater than it is in ordinary collisions because of punitive damages. In theory, robocars should drive the costs down because of the reductions in collisions, and that means savings for the car buyer and for society and thus cheaper auto insurance. However, if the cost per collision is much higher even though the number of collisions drops, there is uncertainty over whether autonomous vehicles will save money for both parties.

California’s Proposition 103 dictates that any insurance policy’s price must be based on weighted factors, and the top 3 weighted factors must be, 1. driving record, 2. number of miles driven and 3. number of years of experience. Other factors like the type of car someone has (i.e. autonomous vehicle) will be weighed lower. Subsequently, this law makes it very hard to get cheap insurance for a robocar.

Nevada Policy: AB 511 Section 8 This short piece of legislation accomplishes the goal of setting good standards for the DMV to follow. By setting general standards (part a), insurance requirements (part b), and safety standards (part c), this sets a precedent for these areas without being too limited with details, leaving them to be decided by the DMV instead of the politicians. …part b only discusses insurance briefly, saying the state must, “Set forth requirements for the insurance that is required to test or operate an autonomous vehicle on a highway within this State.” The definitions set in the second part of Section 8 are not specific enough. Following the open-ended standards set in the earlier part of the Section 8 is good for continuity, but not technically addressing the problem. According to Ryan Calo, Director of Privacy and Robotics for Stanford Law School’s Center for Internet and Society (CIS), the bill’s definition of “autonomous vehicles” is unclear and circular. In the context of this legislation, autonomous driving is seen as a binary system of existence, but in reality, it falls more under a spectrum.

Overall, AB 511 did not address either the technological liabilities and barely mentioned the non-technological liabilities that are necessary to overcome for future success of autonomous vehicles. Since it was the first type of legislation to ever approach the issue of autonomous vehicles, it is understandable that the policymakers did not want to go into specifics and instead rely on future regulation to determine the details.

California Policy: SB 1298…would require the adoption of safety standards and performance requirements to ensure the safe operation and testing of “autonomous vehicles” on California public roads. The bill would allow autonomous vehicles to be operated or tested on the public roads on the condition they meet safety standards and performance requirements of the bill. SB 1298’s 66 lines of text is also considerably longer than AB 511’s 12 lines of relevant text (the entirety of AB 511 is much longer but consists of irrelevant information for the purposes of autonomous cars). would require the adoption of safety standards and performance requirements to ensure the safe operation and testing of “autonomous vehicles” on California public roads. The bill would allow autonomous vehicles to be operated or tested on the public roads on the condition they meet safety standards and performance requirements of the bill. SB 1298’s 66 lines of text is also considerably longer than AB 511’s 12 lines of relevant text (the entirety of AB 511 is much longer but consists of irrelevant information for the purposes of autonomous cars).

SB 1298 has clear intentions to have company developed vehicles by saying in Section 2, Part B that, “autonomous vehicles have been operated safely on public roads in the state in recent years by companies developing and testing this technology” and how these companies have set the standard for what safety standards will be necessary for future testing by others. This part of the legislation implicitly supports Google’s autonomous vehicle because it has the most extensively tested fleet of vehicles out of all the companies, and all this testing has been nearly exclusively done in California. This bill is an improvement over AB 511 by putting more control in the hands of Google to focus on developing the technology, which is a signal by the policymakers to create a climate favorable for Google’s innovation within the constraints of keeping society safe.

To avoid setting a dangerous precedent for liability in accidents, policymakers can consider protecting the car companies from frivolous and malicious lawsuits. Without such legislation, future plaintiffs will be justified to sue Google and put full liability on them. There are also potential free riding effects of the economic moral hazard of putting the blame on the company that makes the technology, not the company that manufactures the vehicle. Since we are assuming that autonomous vehicle technology will all come from one source of Google, then any accident that occurs will pin the blame primarily on Google, the common denominator, not as much as on the car manufacturer…Policy that ensures the costs per accident remains close to today’s current cost will save money for both the insurer and customer. This could potentially mean putting a cap on rewards towards the recipients or punishments towards the company to limit shocks to the industry. Overall, a policymaker can choose to create a gradual limit on the amount of liability placed on the vendor based on certain technology or scaling issues that are met without accidents.

SB 1298 manages to cover some of the shortcomings of AB 511, such as how to improve upon the definition of an autonomous vehicle, as well as looking more towards the future by giving Google more responsibility and alleviating some of the non-technical liability by considering their product “under development”. However, both pieces of legislation fail to address the specific technical liabilities such as bugs in the code base or computer attacks, and non-technical liabilities such as insurance or accident liability.

“Can I See Your License, Registration and C.P.U.?”, Tyler Cowen; see also his “What do the laws against driverless cars look like?”:

The driverless car is illegal in all 50 states. Google, which has been at the forefront of this particular technology, is asking the Nevada legislature to relax restrictions on the cars so it can test some of them on roads there. Unfortunately, the very necessity for this lobbying is a sign of our ambivalence toward change. Ideally, politicians should be calling for accelerated safety trials and promising to pass liability caps if the cars meet acceptable standards, whether that be sooner or later. Yet no major public figure has taken up this cause.

Enabling the development of driverless cars will require squadrons of lawyers because a variety of state, local and federal laws presume that a human being is operating the automobiles on our roads. No state has anything close to a functioning system to inspect whether the computers in driverless cars are in good working order, much as we routinely test emissions and brake lights. Ordinary laws change only if legislators make those revisions a priority. Yet the mundane political issues of the day often appear quite pressing, not to mention politically safer than enabling a new product that is likely to engender controversy.

Politics, of course, is often geared toward preserving the status quo, which is highly visible, familiar in its risks, and lucrative for companies already making a profit from it. Some parts of government do foster innovation, such as DARPA, the Defense Advanced Research Projects Agency, which is part of the Defense Department. DARPA helped create the Internet and is supporting the development of the driverless car. It operates largely outside the public eye; the real problems come when its innovations start to enter everyday life and meet political resistance and disturbing press reports.

…In the meantime, transportation is one area where progress has been slow for decades. We’re still flying 747s, a plane designed in the 1960s. Many rail and bus networks have contracted. And traffic congestion is worse than ever. As I’ argued in a previous column, this is probably part of a broader slowdown of technological advances.

But it’s clear that in the early part of the 20th century, the original advent of the motor car was not impeded by anything like the current mélange of regulations, laws and lawsuits. Potentially major innovations need a path forward, through the current thicket of restrictions. That debate on this issue is so quiet shows the urgency of doing something now.

Ryan Calo of the CIS argues essentially that no specific law bans autonomous cars and the threat of the human-centric laws & regulations is overblown. (See the later Russian incident.)

“SCU conference on legal issues of robocars”, Brad Templeton:

Liability: After a technology introduction where Sven Bieker of Stanford outlined the challenges he saw which put fully autonomous robocars 2 decades away, the first session was on civil liability. The short message was that based on a number of related cases from the past, it will be hard for manufacturers to avoid liability for any safety problems with their robocars, even when the systems were built to provide the highest statistical safety result if it traded off one type of safety for another. In general when robocars come up as a subject of discussion in web threads, I frequently see “Who will be liable in a crash” as the first question. I think it’s a largely unimportant question for two reasons. First of all, when the technology is new, there is no question that any lawsuit over any incident involving the cars will include the vendor as the defendant, in many cases with justifiable reasons, but even if there is no easily seen reason why. So potential vendors can’t expect to not plan for liability. But most of all, the reality is that in the end, the cost of accidents is borne by car buyers. Normally, they do it by buying insurance. But if the accidents are deemed the fault of the vehicle maker, this cost goes into the price of the car, and is paid for by the vehicle maker’s insurance or self-insurance. It’s just a question of figuring out how the vehicle buyer will pay, and the market should be capable of that (though see below.) No, the big question in my mind is whether the liability assigned in any lawsuit will be significantly greater than it is in ordinary collisions where human error is at fault, because of punitive damages…Unfortunately, some liability history points to the latter scenario, though it is possible for statutes to modify this.

Insurance: …Because Prop 103 [specifying insurance by weighted factors, see previous] is a ballot proposition, it can’t easily be superseded by the legislature. It takes a 2/3 vote and a court agreeing the change matches the intent of the original ballot proposition. One would hope the courts would agree that cheaper insurance to encourage safer cars would match the voter intent, but this is a challenge.

Local and criminal laws: The session on criminal laws centered more on the traffic code (which isn’t really criminal law) and the fact it varies a lot from state to state. Indeed, any robocar that wants to operate in multiple states will have to deal with this, though fortunately there is a federal standard on traffic controls (signs and lights) to rely on. Some global standards are a concern - the Geneva convention on traffic laws requires every car has a driver who is in control of the vehicle. However, I think that governments will be able to quickly see - if they want to - that these are laws in need of updating. Some precedent in drunk driving can create problems - people have been convicted of DUI for being in their car, drunk, with the keys in their pocket, because they had clear intent to drive drunk. However, one would hope the possession of a robocar (of the sort that does not need human manual driving) would express an entirely different intent to the law.

“Definition of necessary vehicle and infrastructure systems for Automated Driving”, European Commission report 29 June 2011:

Yet another paramount aspect tightly related to automated driving at present and in the near future, and certainly related to autonomous driving in the long run, is the interpretation of the Vienna Convention. It will be shown in the report how this European legislation is commonly interpreted, how it creates the framework necessary to deploy on a large scale automated and cooperative driving systems, and what legal limitations are foreseen in making the new step toward autonomous driving. The report analyses in the same context other conventions and legislative acts, searches for gaps in the current legislation and makes an interesting link with the aviation industry where several lessons can be learnt from.

It seems appropriate to end this summary with a few remarks not directly related to the subject of this report, but worth in the process of thinking about automated driving, cooperative driving, and autonomous driving. The progress in the human history has systematically taken the path of the shortest resistance and has often bypassed governmental rules, business models, and the obvious thinking. At the end of the 1990s nobody was anticipating the prominent role the smart phone would have in 10 years, but scientists were busy planning journeys to Mars within the same timeframe. The latter has not happened and will probably not happen soon… One lesson humanity has learnt during its existence is that historical changes that followed the path of the minimum resistance triggered at a later stage fundamental changes in the society. “A car is a car” like David Strickland, administrator of the National Highway Traffic Safety Administration (NHTSA) in the U.S. said in his speech at the Telematics Update conference in Detroit, June 2011, but it may drive soon its progress along a historical path of minimum resistance.

An automated driving systems needs to meet the Vienna Convention (see Section 3, aspect 2). The private sector, especially those who are in the end responsible for the performance of the vehicle, should be involved in the discussion.

The Vienna Convention on Road Traffic is an international treaty designed to facilitate international road traffic and to increase road safety by standardizing the uniform traffic rules among the contracting parties. This convention was agreed upon at the United Nations Economic and Social Council’s Conference on Road Traffic (October 7, 1968 - November 8, 1968). It came into force on May 21 1977. Not all EU countries have ratified the treaty, see Figure 13 (e.g. Ireland, Spain and UK did not). It should be noted that in 1968, animals were still used for traction of vehicles and the concept of autonomous driving was considered to be science fiction. This is important when interpreting the text of the treaty: in a strict interpretation to the letter of the text, or interpretation of what is meant (at that time).

The common opinion of the expert panel is that the Vienna Convention will have only a limited effect on the successful deployment of automated driving systems due to several reasons:

OEMs already deal with the situation that some of the Advanced Driver Assistance Systems touch the Vienna Convention today. For example, they provide an on/off switch for ADAS or allow an overriding of the functions by the driver. They develop their ADAS in line with the RESPONSE Code of Practice (2009) [41] following the principle that the driver is in control and remains responsible. In addition, the OEMs have a careful marketing strategy and they do not exaggerate and do not claim that an ADAS is working in all driving situations or that there is a solution to “all” safety problems.

Automation is not black and white, automated or not automated, but much more complex, involving many design dimensions. A helpful model of automation is to consider different levels of assistance and automation that can e.g. be organized on a 1d- scale [42]. Several levels could be within the Vienna Convention, while extreme levels are outside of today’s version of the Vienna Convention. For example, one partitioning could be to have levels of automation Manual, Assisted, Semi-Automated, Highly Automated, and Fully Automated driving, see Figure 14. In highly automated driving, the automation has the technical capabilities to drive almost autonomously, but the driver is still in the loop and able to take over control when it is necessary. Fully automated driving like PRT, where the driver is not required to monitor the automation and does not have the ability to take over control, seems not to be covered by the Vienna Convention.

Criteria for deciding if the automation is still in line with the Vienna Convention could be:

the involvement of the driver in the driving task (vehicle control),

the involvement of the driver in monitoring the automation and the traffic environment,

the ability to take over control or to override the automation

The Vienna Convention already contains openings, or is variable, or can be changed.

It contains a certain variability regarding the autonomy in the means of transportation, e.g. “to control the vehicle or guide the animals”. It is obvious that some of the current technological developments were not foreseen by the authors of the Vienna Convention. Issues like platooning are not addressed. The Vienna Convention already contains in Annex 5 (chapter 4, exemptions) an opening to be investigated with appropriate legal expertise:

“For domestic purposes, Contracting Parties may grant exemptions from the provisions of this Annex in respect of: (c) Vehicles used for experiments whose purpose is to keep up with technical progress and improve road safety; (d) Vehicles of a special form or type, or which are used for particular purposes under special conditions”. - In addition, the Vienna Convention can be changed. The last change was made in 2006. A new paragraph (paragraph 6) was added to Article 8 stating that the driver should minimize any activity other than driving.

…different understandings of the term “to control” with no clear consensus [44]: 1. Control in a sense of influencing e.g. the driver controls the vehicle movements, the driver can override the automation and/or the driver can switch the automation off. 2. Control in a sense of monitoring e.g. the driver monitors the actions of the automation. Both interpretations allow the use of some form of automation in a vehicle as it can be seen in today’s cars where e.g. ACC or emergency brake assistance systems etc. are available.

The first interpretation allows automation that can be overridden by the driver or that reacts in emergency situations only when the driver cannot cope with the situation anymore. Forms of automation that cannot be overridden seem to be not in line with the first interpretation [45, p. 818]. The second interpretation is more flexible and would allow also forms of automation that cannot be overridden and are within the Vienna Convention as long as the driver monitors the automation [44]. …In the literature, some other assistance and automation functions were appraised by juridical experts. For example, [46] postulates that automatic emergency braking systems are in line with the Vienna Convention as long as they react only when a crash is unavoidable (collision mitigation). Otherwise a conflict between the driver’s intention (here, steering) and the reaction of the automation (here, braking) cannot be excluded. Albrecht [47] concludes that an Intelligent Speed Adaptation (ISA) which cannot be overridden by the driver is not in line with the Vienna Convention because it is not consistent with Article 8 and Article 13 of the Vienna Convention.

…As soon as data from the vehicle is used for V2X-communication or is stored in the vehicle itself, data protection and privacy issues become relevant. Directives and documents that need to be checked include:

Directive 95/46/EC on the protection of individuals with regard to the processing of personal data and on the free movement of such data;

Directive 2010/40/EU on the framework for the deployment of Intelligent Transport Systems in the field of road transport and for interfaces with other modes of transport;

WP 29 Working document on data protection and privacy implications in the eCall initiative and the European Data Protection Supervisor (EDPS) opinion on ITS Action Plan and Directive.

The bottleneck is that at the current stage of development the risk related costs and benefits of viable deployment paths are unknown, combined with the fact that the deployment paths themselves are wide open because the possible deployment scenarios are not assessed and debated in a political environment. There is currently no consensus amongst stakeholders on which of the deployment scenarios proposed will eventually prevail…Changes in EU legislation might change the role of players and increase the risk for them. Any change in EU legislation will change the position of the players, and uncertainty in which direction this change (gap) would go adds to the risk. This prohibits players from having an outspoken opinion on the issue. If an update of existing legislation is considered, this should be European legislation, not national legislation. It would be better to go for a world-wide harmonized legislation, when it is decided to take that path.

A useful case study for understanding the issues associated with automated driving can be found in SAFESPOT [4] which can be viewed as a parallel to automated driving functions (for more details, see Appendix I. Related to aspect 3). SAFESPOT provided an in-depth analysis of the legal aspects of the service named ‘Speed Warning’, in two configurations V2I and V2V. It is performed against two fundamentally different law schemes, namely Dutch and English law. This analysis concluded that the concept of co-operative systems raises questions and might complicate legal disputes. This is for several reasons:

There are more parties involved, all with their own responsibilities for the proper functioning of elements of a cooperative system.

Growing technical interdependencies between vehicles, and between vehicles and the infrastructure, may also lead to system failure, including scenarios that may be characterised as an unlucky combination of events (“a freak accident”) or as a failure for which the exact cause simply cannot be traced back (because of the technical complexity).

Risks that cannot be influenced by the people who suffer the consequences tend to be judged less acceptable by society and, likewise, from a legal point of view.

The in-depth analysis of SAFESPOT concluded that (potential) participants such as system producers and road managers may well be exposed to liability risks. Even if the driver of the probe vehicle could not successfully claim a defense (towards other road users), based on a failure of a system, system providers and road managers may still remain (partially) responsible through the mechanism of subrogation and right of recourse.

Current law states that the driver must be in control of his vehicle at all times. In general, EU drivers are prohibited to exhibit dangerous behaviour while driving. The police have prosecuted drivers in the UK for drinking and/or eating; i.e. only having one hand on the steering wheel. The use of a mobile phone while driving is prohibited in many European countries, only use of phones equipped for hands free operation are permitted. Liability still rests firmly with the driver for the safe operation of vehicles.

New legislation may be required for automated driving. It is highly unlikely that any OEM or supplier will risk introducing an automatic driving vehicle (where responsibility for safe driving is removed from the driver) without there being a framework of new legislation which clearly sets out where their responsibility and liability begins and ends. In some ways it could be seen as similar to warranty liability, the OEM warrants certain quality and performance levels, backed by reciprocal agreements within the supply chain. Civil (and possibly criminal) liability in the case of accidents involving automated driving vehicles is a major issue that can truly delay the introduction of these technologies…Since there are no statistical records of the effects of automated driving systems, the entrepreneurship of insurers should compensate for the issue of unknown risks…The following factors are regarded as hindering an optimal role to be played by the insurance industry in promoting new safety systems through their insurance policies:

Premium-setting is based on statistical principles, resulting in a time-lag problem;

Competition/sensitive relationships with clients;

Investment costs (e.g. aftermarket installations);

Administrative costs;

Market regulation

No precedence lawsuits of liability with automated systems have happened to date. The Toyota malfunctions of their brake-by-wire system in 2010 did not end in a lawsuit. A system like parking assist is technically not redundant. What would happen if the driver claimed he/she could not override the brakes? For (premium) insurance a critical mass is required, so initially all stake-holders including governments should potentially play a role.

“Automotive Autonomy: Self-driving cars are inching closer to the assembly line, thanks to promising new projects from Google and the European Union”, Wright 2011:

The Google project has made important advances over its predecessor, consolidating down to one laser rangefinder from five and incorporating data from a broader range of sources to help the car make more informed decisions about how to respond to its external environment. “The threshold for error is minuscule,” says Thrun, who points out that regulators will likely set a much higher bar for safety with a self-driving car than for one driven by notoriously error-prone humans.

“The future of driving, Part III: hack my ride”, Lee 2008:

Of course, one reason that private investors might not want to invest in automotive technologies is the risk of excessive liability in the case of crashes. The tort system serves a valuable function by giving manufacturers a strong incentive to make safe, reliable products. But too much tort liability can have the perverse consequence of discouraging the introduction of even relatively safe products into the marketplace. Templeton tells Ars that the aviation industry once faced that problem. At one point, “all of the general aviation manufacturers stopped making planes because they couldn’t handle the liability. They were being found slightly liable in every plane crash, and it started to cost them more than the cost of manufacturing the plane.” Airplane manufacturers eventually convinced Congress to place limits on their liability. At the moment, crashes tend to lead to lawsuits against human drivers, who rarely have deep pockets. Unless there is evidence that a mechanical defect caused the crash, car manufacturers tend not to be the target of most accident-related lawsuits. That would change if cars were driven by software. And because car manufacturers have much deeper pockets than individual drivers do, plaintiffs are likely to seek much larger damages than they would against human drivers. That could lead to the perverse result that even safer self-driving cars would be more expensive to insure than human drivers. Since car manufacturers, rather than drivers, would be the first ones sued in the event of an accident, car companies are likely to protect themselves by buying their own insurance. And if insurance premiums get too high, they may take the route the aviation industry did and seek limits on liability. An added benefit for consumers is that most would never have to worry about auto insurance. Cars would come preinsured for the life of the vehicle (or at least the life of the warranty)…Self-driving vehicles will sit at the intersection of two industries that are currently subject to very different regulatory regimes. The automobile industry is heavily regulated, while the software industry is largely unregulated at all. The most fundamental decision regulators will need to make is whether one of these existing regulatory regimes will be suitable for self-driving technologies, or whether an entirely new regulatory framework will be needed to accommodate them.

http://www.917wy.com/topicpie/2008/11/future-of-driving-part-3/2

It’s inevitable that at some point, a self-driving vehicle will be involved in a fatal crash which generates worldwide publicity. Unfortunately, even if self-driving vehicles have amassed an overall safety record that’s superior to that of human drivers, the first crash is likely to prompt calls for drastic restrictions on the use of self-driving technologies. It will therefore be important for business leaders and elected officials to lay the groundwork by both educating the public about the benefits of self-driving technologies and managing expectations so that the public isn’t too surprised when crashes happen. Of course, if the first self-driving cars turn out to be significantly less safe than the average human driver, then they should be pulled off the streets and re-tooled. But this seems unlikely to happen. A company that introduced self-driving technology into the marketplace before it was ready would not only have trouble convincing regulators that its cars are safe, but it would be risking ruinous lawsuits, as well. The far greater danger is that the combination of liability fears and red tape will cause the United States to lose the initiative in self-driving technologies. Countries such as China, India, and Singapore that have more autocratic regimes or less-developed economies may seize the initiative and introduce self-driving cars while American policymakers are still debating how to regulate them. Eventually, the specter of other countries using technologies that aren’t available in the United States will spur American politicians into action, but only after several thousand Americans lose their lives unnecessarily at the hands of human drivers.

…One likely area of dispute is whether people will be allowed to modify the software on their own cars. The United States has a long tradition of people tinkering with both their cars and their computers. No doubt, there will be many people who are interested in modifying the software on their self-driving cars. But there is likely to be significant pressure for legislation criminalizing unauthorized tinkering with self-driving car software. Both car manufacturers and (as we’ll see shortly) the law enforcement community are likely to be in favor of criminalizing the modification of car software. And they’ll have a plausible safety argument: buggy car software would be dangerous not only to the car owner but to others on the road. The obvious analogy is to the DMCA, which criminalized unauthorized tinkering with copy protection schemes. But there are also important differences. One is that car manufacturers will be much more motivated to prevent tinkering than Apple or Microsoft are. If manufacturers are liable for the damage done by their vehicles, then tinkering not only endangers lives, but their bottom lines as well. It’s unlikely that Apple would ever sue people caught jailbreaking their iPhones. But car manufacturers probably will contractually prohibit tinkering and then sue those caught doing it for breach of contract.

http://www.917wy.com/topicpie/2008/11/future-of-driving-part-3/3

The more stalwart advocate of locked-down cars is likely to be the government, because self-driving car software promises to be a fantastic tool for social control. Consider, for example, how useful locked-down cars could be to law enforcement. Rather than physically driving to a suspect’s house, knocking on his door (or not), and forcibly restraining, handcuffing, and escorting a suspect to the station, police will be able to simply seize a suspect’s self-driving car remotely and order it to drive to the nearest police station. And that’s just the beginning. Locked-down car software could be used to enforce traffic laws, to track and log peoples’ movements for later review by law enforcement, to enforce curfews, to clear the way for emergency vehicles, and dozens of other purposes. Some of these functions are innocuous. Others will be very controversial. But all of them depend on restricting user control over their own vehicles. If users were free to swap in custom software, they might disable the government’s “back door” and re-program it to ignore government requirements. So the government is likely to push hard for laws mandating that only government-approved software run self-driving cars.

…It’s too early to say exactly what the car-related civil liberties fights will be about, or how they will be resolved. But one thing we can say for certain is that the technical decisions made by today’s computer scientists will be important for setting the stage for those battles. Advocates for online free speech and anonymity have been helped tremendously by the fact that the Internet was designed with an open, decentralized architecture. The self-driving cars of the future are likely to be built on top of software tools that are being developed in today’s academic labs. By thinking carefully about the ways these systems are designed, today’s computer scientists can give tomorrow’s civil liberties their best shot at preserving automotive freedom.

http://www.917wy.com/topicpie/2008/11/future-of-driving-part-3/4

In our interview with him, Congressman Adam Schiff described the public’s perception of autonomous driving technologies as a reflection of his own reaction to the idea: one that is a mixture of both fascination and skepticism. Schiff explained that the public’s fascination comes from amazement at how advanced this technology already has become, plus with Google’s sponsorship and endorsement it becomes even more alluring.

Skepticism of autonomous vehicle technologies comes from a missing element of trust. According to Clifford Nass, a professor of communications and sociology at Stanford University, this trust is an aspect of public opinion that must be earned through demonstration more so than through use. When people see a technology in action, they will begin to trust it. Professor Nass specializes in studying the way in which human beings relate to technology, and he has published several books on the topic including The Man Who Lied to His Laptop: What Machines Teach Us About Human Relationships. In our interview with him, Professor Nass explained that societal comfort with technology is gained through experience, and acceptance occurs when people have seen a technology work enough times collectively. He also pointed out that it took a long time for people to develop trust in air transportation, something that we almost take for granted now. It is certainly not the case that autonomous cars need to be equivalent in safety to plane flight before the public would adopt them. However, as Noel du Toit pointed out, we have a higher expectation for autonomous cars than we do for ourselves. Simply put, if we are willing to relinquish the “control” over our vehicles to an autonomous power, it will likely have to be under the condition that the technology drives more adeptly than we ever possibly could. Otherwise, there will simply be no trusting it. Interestingly, du Toit brought up a recent botched safety demonstration by Volvo in May of 2010. In the demonstration, Volvo showcased to the press how its emergency braking system works as part of an “adaptive cruise control” system. These systems allow a driver to set both a top speed and a following distance, which the vehicle then automatically maintains. As a consequence, if the preceding vehicle stops short, the system acts as the foundation for an emergency-braking maneuver. However, In Volvo’s demonstration the car smashed directly into a trailer13. Even though the system worked fine in several cases during the day’s worth of demonstrations, video of that one mishap went viral and did little to help the public gain trust in the technology.

Calo pointed out at that future issues related to autonomous vehicles would be approached from a standpoint of “negative liabilities”, meaning that we can assume something is legal unless there exist explicit laws against it. This discussion also led to the concept of what a driverless car would look like to bystanders, and the kind of panic that might garner. A real-life example of this occurred in Moscow during the VisLab van trek to Shanghai11. In this case, an autonomous electric van was stopped by Russian authorities due to its apparent lack of a driver behind the wheel. Thankfully, engineers present were able to convince the Russian officer who stopped the vehicle not to issue a ticket. The above [Nevadan] legislation fits in well with the information that we collected from Congressman Schiff about potential federal involvement in autonomous vehicle technology. Basically, Schiff relayed the idea that a strong governmental role expected for this technology would come in the form of regulating safety. Furthermore, he called attention to hefty governmental requirements for crash testing that every new vehicle must meet before it is allowed on the road.

In autonomous driving, liability concerns can be inferred through a couple of examples. In one example, Noel du Toit described DARPA’s use of hired stunt drivers to share the testing grounds with driverless vehicle entries in the 2007 Urban Challenge. This behavior clearly illustrates the level of precaution that the DARPA officials felt it was necessary to take. In another example, Dmitri Dolgov expounded on how Google’s cars are never driving by themselves; whenever they are operated on public roads, there are at least two well-trained operators in the car. Dolgov went on to say that these operators “are in control at all times”, which helps illustrate Google’s position-they are not taking any chances when it comes to liabilities. Kent Kresa, former CEO of Northrup Grumman and interim chairman of GM in 2009, was also concerned about the liability issues presented by autonomous vehicles. Kresa felt that a future with driverless cars piloting the streets was somewhat unimaginable at present, especially when one considers the possibility of a pedestrian getting hit. In the case of such a collision it is still very unclear who would be at fault. Whether or not the company that made the vehicle would be responsible is at present unknown.

A conversation we had with Bruce Gillman, the public information officer for the Los Angeles Department of Transportation (DOT), revealed that the department is very busy putting out many other fires. Gillman noted that DOT is focused on getting people out of their cars and onto bikes or into buses. Thus, autonomous vehicles are not on their radar. Moreover, Gillman was adamant that DOT would wait until autonomous vehicles were being manufactured commercially before addressing any issues concerning them. His viewpoint certainly reinforces that idea that supportive infrastructure updates coming form a city government level would be unlikely. No matter what adoption pathway is used, federal government financial support could come in the form of incentives and subsidies like those seen during the initial rollout of hybrid vehicles. However, Brian Thomas explained that this would only be possible if the federal government was willing to do a cost-benefit valuation for the mainstream introduction of autonomous vehicles.

http://www.pickar.caltech.edu/e103/Final%20Exams/Autonomous%20Vehicles%20for%20Personal%20Transport.pdf [shades of Amara’s law: we always overestimate in the short run & underestimate in the long run]

Car manufacturers might be held liable for a larger share of the accidents-a responsibility they are certain to resist. (A legal analysis by Nidhi Kalra and her colleagues at the RAND Corporation suggests this problem is not insuperable.) –“Leave the Driving to It”, Brian Hayes American Scientist 2011

The RAND report: “Liability and Regulation of Autonomous Vehicle Technologies”, Kalra et al 2009:

In this work, we first evaluate how the existing liability regime would likely assign responsibility in crashes involving autonomous vehicle technologies. We identify the controlling legal principles for crashes involving these technologies and examine the implications for their further development and adoption. We anticipate that consumer education will play an important role in reducing consumer overreliance on nascent autonomous vehicle technologies and minimizing liability risk. We also discuss the possibility that the existing liability regime will slow the adoption of these socially desirable technologies because they are likely to increase liability for manufacturers while reducing liability for drivers. Finally, we discuss the possibility of federal preemption of state tort suits if the U.S. Department of Transportation (US DOT) promulgates regulations and some of the implications of eliminating state tort liability. Second, we review the existing literature on the regulatory environment for autonomous vehicle technologies. To date, there are no government regulations for these technologies, but work is being done to develop initial industry standards.

…Additionally, for some systems, the driver is expected to intervene when the system cannot control the vehicle completely. For example, if a very rapid stop is required, ACC may depend on the driver to provide braking beyond its own capabilities. ACC also does not respond to driving hazards, such as debris on the road or potholes-the driver is expected to intervene. Simultaneously, research suggests that drivers using these conveniences often become complacent and slow to intervene when necessary; this behavioral adaptation means drivers are less responsive and responsible than if they were fully in control (Rudin-Brown and Parker, 2004). Does such evidence suggest that manufacturers may be responsible for monitoring driver behavior as well as vehicle behavior? Some manufacturers have already taken a step toward ensuring that the driver assumes responsibility and is attentive, by requiring the driver to periodically depress a button or by monitoring the driver by sensing eye movements and grip on the steering wheel. As discussed later, litigation may occur around the issue of driver monitoring and the danger of the driver relying on the technology for something that it is not designed to accomplish.

…Ayers (1994) surveyed a range of emerging autonomous vehicle technologies and automated highways, evaluated the likelihood of a shift in liability occurring, discussed the appropriateness of government intervention, and highlighted the most-promising interventions for different technologies. Ayers found that collision warning and collision-avoidance systems “are likely to generate a host of negligence suits against auto manufacturers” and that liability disclaimers and federal regulations may be the most effective methods of dealing with the liability concerns (p. 21). The report was written before many of these technologies appeared on the market, and Ayers further speculated that “the liability for almost all accidents in cars equipped with collision-avoidance systems would conceivably fall on the manufacturer” (p. 22), which could “delay or even prevent the deployment of collision warning systems that are cost-effective in terms of accident reduction” (p. 25). Syverud (1992) examines the legal cases stemming from the introduction of air bags, antilock brakes, cruise control, and cellular telephones to provide some general lessons for the liability concerns for autonomous vehicle technologies. In another report, Syverud (1993) examines the legal barriers to a wide range of IVHSs and finds that liability poses a significant barrier particularly to autonomous vehicle technologies that take control of the vehicle. In this work, Syverud’s interviews with manufacturers reveal that liability concerns had already adversely affected research and development in these technologies in several companies. One interviewee is quoted as saying that “IVHS will essentially remain ‘information technology and a few pie-in-the sky pork barrel control technology demonstrations, at least in this country, until you lawyers do something about products liability law’” (1993, p. 25).

…While the victims in these circumstances could presumably sue the vehicle manufacturer, products liability lawsuits are more expensive to bring and take more time to resolve than run-of-the-mill automobile-crash litigation. This shift in responsibility from the driver to the manufacturer may make no-fault automobile-insurance regimes more attractive. They are designed to provide compensation to victims relatively quickly, and they do not depend upon the identification of an “at-fault” party

…Suppose that autonomous vehicle technologies are remarkably effective at virtually eliminating minor crashes caused by human error. But it may be that the comparatively few crashes that do occur usually result in very serious injuries or fatalities (e.g., because autonomous vehicles are operating at much higher speeds or densities). This change in the distribution of crashes may affect the economics of insuring against them. Actuarially, it is much easier for an insurance company to calculate the expected costs of somewhat common small crashes than of rarer, much larger events. This may limit the downward trend in automobile-insurance costs that we would otherwise expect.

…Suppose that most cars brake automatically when they sense a pedestrian in their path. As more cars with this feature come to be on the road, pedestrians may expect that cars will stop, in the same way that people stick their limbs in elevator doors confident that the door will automatically reopen. The general level of pedestrian care may decline as people become accustomed to this common safety feature. But if there were a few models of cars that did not stop in the same way, a new category of crashes could emerge. In this case, should pedestrians who wrongly assume that a car would automatically stop and are then injured be able to recover? To allow recovery in this instance would seem to undermine incentives for pedestrians to take efficient care. On the other hand, allowing the injured pedestrian to recover may encourage the universal adoption of this safety feature. Since negligence is defined by unreasonableness, the evolving set of shared assumptions about the operation of the roadways-what counts as “reasonable”-will determine liability. Fourth, we think that it is not likely that operators of partially or fully autonomous vehicles will be found strictly liable with driving such vehicles as an ultrahazardous activity. As explained earlier, these technologies will be introduced incrementally and will initially serve merely to aid the driver rather than take full control of the vehicle. This will give the public and courts time to become familiar with the capabilities and limits of the technology. As a result, it seems unlikely that courts will consider its gradual introduction and use to be ultrahazardous. On the other hand, this would not be true if a person attempted to operate a car fully autonomously before the technology adequately matured. Suppose, for example, that a home hobbyist put together his own autonomous vehicle and attempted to operate it on public roads. Victims of any crashes that resulted may well be successful in convincing a court to find the operator strictly liable on the grounds that such activity was ultrahazardous.

…Product-liability law can be divided into theories of liability and kinds of defect. Theories of liability include negligence, misrepresentation, warranty, and strict liability.22 Types of defect include manufacturing defects, design defects, and warning defects. A product-liability lawsuit will involve one or more theories of manufacturer liability attached to a specific allegation of a type of defect. In practice, the legal tests for the theories of liability often overlap and, depending on the jurisdiction, may be identical. … While it is difficult to generalize, automobile (and subsystem) manufacturers may fare well under a negligence standard that uses a cost-benefit analysis that includes crashes avoided from the use of autonomous vehicle technologies. Automakers can argue that the overall benefits from the use of a particular technology outweigh the risks. The number of crashes avoided by the use of these technologies is probably large. …Unfortunately, the socially optimal liability rule is unclear. Permitting the defendant to include the long-run benefits in the cost-benefit analysis may encourage the adoption of technology that can indeed save many lives. On the other hand, it may shield the manufacturer from liability for shorter-run decisions that were inefficiently dangerous. Suppose, for example, that a crash-prevention system operates successfully 70% of the time but that, with additional time and work, it could have been designed to operate successfully 90% of the time. Then suppose that a victim is injured in a crash that would have been prevented had the system worked 90% of the time. Assume that the adoption of the 70-percent technology is socially desirable but the adoption of the 90-percent technology would be even more socially desirable. How should the cost-benefit analysis be conducted? Is the manufacturer permitted to cite the 70% of crashes that were prevented in arguing for the benefits of the technology? Or should the cost-benefit analysis focus on the manufacturer’s failure to design the product to function at 90-percent effectiveness? If the latter, the manufacturer might not employ the technology, thereby leading to many preventable crashes. In calculating the marginal cost of the 90-percent technology, should the manufacturer be able to count the lives lost in the delay in implementation as compared to possible release of the 70-percent technology? …Tortious misrepresentation may play a role in litigation involving crashes that result from autonomous vehicle technologies. If advertising overpromises the benefits of these technologies, consumers may misuse them. Consider the following hypothetical scenario. Suppose that an automaker touts the “autopilot-like” features of its ACC and lane-keeping function. In fact, the technologies are intended to be used by an alert driver supervising their operation. After activating the ACC and lane-keeping function, a consumer assumes that the car is in control and falls asleep. Due to road resurfacing, the lane-keeping function fails, and the automobile leaves the roadway and crashes into a tree. The consumer then sues the automaker for tortious misrepresentation based on the advertising that suggested that the car was able to control itself.

…Finally, it is also possible that auto manufacturers will be sued for failing to incorporate autonomous vehicle technologies in their vehicles. While absence of available safety technology is a common basis for design- defect lawsuits (e.g., Camacho v. Honda Motor Co., 741 P.2d 1240, 1987, overturning summary dismissal of suit alleging that Honda could easily have added crash bars to its motorcycles, which would have prevented the plaintiff’s leg injuries), this theory has met with little success in the automotive field because manufacturers have successfully argued that state tort remedies were preempted by federal regulation (Geier v. American Honda Motor Co., 529 U.S. 861, 2000, finding that the plaintiff’s claim that the manufacturer was negligent for failing to include air bags was implicitly preempted by the National Traffic and Motor Vehicle Safety Act). We discuss preemption and the relationship between regulation and tort in Section 4.3.

…Preemption has arisen in the automotive context in litigation over a manufacturer’s failure to install air bags. In Geier v. American Honda Motor Co. (2000), the U.S. Supreme Court found that state tort litigation over a manufacturer’s failure to install air bags was preempted by the National Traffic and Motor Vehicle Safety Act (Pub. L. No. 89-563). More specifically, the Court found that the Federal Motor Vehicle Safety Standard (FMVSS) 208, promulgated by the US DOT, required manufacturers to equip some but not all of their 1987 vehicle-year vehicles with passive restraints. Because the plaintiffs’ theory that the defendants were negligent under state tort law for failing to include air bags was inconsistent with the objectives of this regulation (FMVSS 208), the Court held that the state lawsuits were preempted. Presently, there has been very little regulation promulgated by the US DOT with respect to autonomous vehicle technologies. Should the US DOT promulgate such regulation, it is likely that state tort law claims that were found to be inconsistent with the objective of the regulation would be held to be preempted under the analysis used in Geier. Substantial litigation might be expected as to whether particular state-law claims are, in fact, inconsistent with the objectives of the regulation. Resolution of those claims will depend on the specific state tort law claims, the specific regulation, and the court’s analysis of whether they are “inconsistent.” …Our analysis necessarily raises a more general question: Why should we be concerned about liability issues raised by a new technology? The answer is the same as for why we care about tort law at all: that a tort regime must balance economic incentives, victim compensation, and corrective justice. Any new technology has the potential to change the sets of risks, benefits, and expectations that tort law must reconcile. …Congress could consider creating a comprehensive regulatory regime to govern the use of these technologies. If it does so, it should also consider preempting inconsistent state-court tort remedies. This may minimize the number of inconsistent legal regimes that manufacturers face and simplify and speed the introduction of this technology. While federal preemption has important disadvantages, it might speed the development and utilization of this technology and should be considered, if accompanied by a comprehensive federal regulatory regime.

…This tension produced “a standoff between airbag proponents and the automakers that resulted in contentious debates, several court cases, and very few airbags” (Wetmore, 2004, p. 391). In 1984, the US DOT passed a ruling requiring vehicles manufactured after 1990 to be equipped with some type of passive restraint system (e.g., air bags or automatic seat belts) (Wetmore, 2004); in 1991, this regulation was amended to require air bags in particular in all automobiles by 1999 (Pub. L. No. 102-240). The mandatory performance standards in the FMVSS further required air bags to protect an unbelted adult male passenger in a head-on, 30 mph crash. Additionally, by 1990, the situation had changed dramatically, and air bags were being installed in millions of cars. Wetmore attributes this development to three factors: First, technology had advanced to enable air-bag deployment with high reliability; second, public attitude shifted, and safety features became important factors for consumers; and, third, air bags were no longer being promoted as replacements but as supplements to seat belts, which resulted in a sharing of responsibility between manufacturers and passengers and lessened manufacturers’ potential liability (Wetmore, 2004). While air bags have certainly saved many lives, they have not lived up to original expectations: In 1977, NHTSA estimated that air bags would save on the order of 9,000 lives per year and based its regulations on these expectations (Thompson, Segui-Gomez, and Graham, 2002). Today, by contrast, NHTSA calculates that air bags saved 8,369 lives in the 14 years between 1987 and 2001 (Glassbrenner, undated). Simultaneously, however, it has become evident that air bags pose a risk to many passengers, particularly smaller passengers, such as women of small stature, the elderly, and children. NHTSA (2008a) determined that 291 deaths were caused by air bags between 1990 and July 2008, primarily due to the extreme force that is necessary to meet the performance standard of protecting the unbelted adult male passenger. Houston and Richardson (2000) describe the strong reaction to these losses and a backlash against air bags, despite their benefits. The unintended consequences of air bags have led to technology developments and changes to standards and regulations. Between 1997 and 2000, NHTSA developed a number of interim solutions designed to reduce the risks of air bags, including on-off switches and deployment with less force (Ho, 2006). Simultaneously, safer air bags, called advanced air bags, were developed that deploy with a force tailored to the occupant by taking into account the seat position, belt usage, occupant weight, and other factors. In 2000, NHTSA mandated that the introduction of these advanced air bags begin in 2003 and that, by 2006, every new passenger vehicle would include these safety measures (NHTSA, 2000). What lessons does this experience offer for regulation of autonomous vehicle technologies? We suggest that modesty and flexibility are necessary. The early air-bag regulators envisioned air bags as being a substitute for seat belts because the rates of seat-belt usage were so low and appeared intractable. Few anticipated that seat-belt usage would rise as much over time as it has and that air bags would eventually be used primarily as a supplement rather than a substitute for seat belts. Similarly unexpected developments are likely to arise in the context of autonomous vehicle technologies. In 2006, for example, Honda introduced its Accord model in the UK with a combined lane-keeping and ACC system that allows the vehicle to drive itself under the driver’s watch; this combination of features has yet to be introduced in the United States (Miller, 2006). Ho (2006, p. 27) observes a general trend that “the U.S. market trails Europe, and the European market trails Japan by 2 to 3 years.” What is the extent of these differences? What aspects of the liability and regulatory rules in those countries have enabled accelerated deployment? What other factors are at play (e.g., differences in consumers’ sensitivity to price)?

“New Technology - Old Law: Autonomous Vehicles and California’s Insurance Framework”, Peterson 2012:

This Article will address this issue and propose ways in which auto insurance might change to accommodate the use of AVs. Part I briefly reviews the background of insurance regulation nationally and in California. Part II discusses general insurance and liability issues related to AVs. Part III discusses some challenges that insurers and regulators may face when setting rates for AVs, both generally and under California’s more idiosyncratic regulatory structure. Part IV discusses challenges faced by California insurers who may want to reduce rates in a timely way when technological improvements rapidly reduce risk.

…When working within the context of a file-and-use or use-and-file environment, AVs will present only modest challenges to an insurer that wants to write these policies. The main challenge will arise from the fact that the policy must be rated for a new technology that may have an inadequate base of experience for an actuary to estimate future losses.21 “Prior approval” states, like California, require that automobile rates be approved prior to their use in the marketplace.22 These states rely more on regulation than on competition to modulate insurance rates.23 In California, automobile insurance rates are approved in a two-step process. The first step is the creation of a “rate plan.”24 The rate plan considers the insurer’s entire book of business in the relative line of insurance and asks the question: How much total premium must the insurer collect in order to cover the projected risks, overhead and permitted profit for that line?25 The insurer then creates a “class plan.” The class plan asks the question: How should different policyholders’ premiums be adjusted up or down based on the risks presented by different groups or classes of policyholders?26 Among other factors, the Department of Insurance requires that the rating factors comply with California law and be justified by the loss experience for the group.27 Rating a new technology with an unproven track record may include a considerable amount of guesswork. …California is the largest insurance market in the United States, and it is the sixth largest among the countries of the world.28 Cars are culture in this most populous state. There are far more insured automobiles in California than any other state.29

…Although adopted by the barest majority, [California’s] Proposition 103 [see previous discussion of its 3-part requirement for rating insurance premiums] may be amended by the legislature only by a two-thirds vote, and then only if the legislation “further[s] [the] purposes” of Proposition 103.68 Thus, Proposition 103 and the regulations adopted by the Department of Insurance are the matrix in which most (but not all) insurance is sold and regulated in California.69 …The most sensible approach to this dilemma, at least with respect to AVs, would be to abolish or substantially re-order the three mandatory rating factors. However, this is more easily said than done. As noted above, amending Proposition 103 requires a two-thirds vote of the legislature.160 Moreover, section 8(b) of the Proposition provides: “The provisions of this act shall not be amended by the Legislature except to further its purposes.”161 Both of these requirements can be formidable hurdles. Persistency discounts serve as an example. Most are aware that their insurer discounts their rates if they have been with the insurer for a period of time.162 This is called the “persistency discount.” The discount is usually justified on the basis that persistency saves the insurer the producing expenses associated with finding a new insured. If one wants to change insurers, Proposition 103 does not permit the subsequent insurer to match the persistency discount offered by the insured’s current insurer.163 Thus, the second insurer could not compete by offering the same discount. Changing insurers, then, was somewhat like a taxable event. The “tax” is the loss of the persistency discount when purchasing the new policy. The California legislature concluded that this both undermined competition and drove up the cost of insurance by discouraging the ability to shop for lower rates. …Despite these legislative findings, the Court of Appeal held the amendment invalid because, in the Court’s view, it did not further the purposes of Proposition 103.165 The Court also held that Proposition 103 vests only the Insurance Commissioner with the power to set optional rating factors.166 Thus, the legislature, even by a super majority, may not be authorized to adopt rating factors for auto insurance. Following this defeat in the courts, promoters of “portable persistency” qualified a ballot initiative to amend this aspect of Proposition 103. With a vote of 51.9% to 48.1%, the initiative failed in the June 8, 2010 election.167

…The State of Nevada recently adopted regulations for licensing the testing of AVs in the state. The regulations would require insurance in the minimum amounts required for other cars “for the payment of tort liabilities arising from the maintenance or use of the motor vehicle.”73 The regulation, however, does not suggest how the tort liability may arise. If there is no fault on the part of the operator or owner, then liability may arise, if at all, only for the manufacturer or supplier. Manufacturers and suppliers are not “insureds” under the standard automobile policy-at least so far. Thus, for the reasons stated above, owners, manufacturers and suppliers may fall outside the coverage of the policy.

…One possible approach would be to invoke the various doctrines of products liability law. This would attach the major liability to sellers and manufacturers of the vehicle. However, it is doubtful that this is an acceptable approach for several reasons. For example, while some accidents are catastrophic, fortunately most accidents cause only modest damages. By contrast, products liability lawsuits tend to be complex and expensive. Indeed, they may require the translation of hundreds or thousands of engineering documents-perhaps written in Japanese, Chinese or Korean…See In re Puerto Rico Electric Power Authority, 687 F.2d 501, 505 (1st Cir. 1982) (stating each party to bear translation costs of documents requested by it but cost possibly taxable to prevailing party). Translation costs of Japanese documents in range of $250,000, and translation costs of additional Spanish documents may exceed that amount.

…Commercial insurers of manufacturers and suppliers are not encumbered with Proposition 103’s unique automobile provisions,197 therefore they need not offer a GDD, nor need they conform to the ranking of the mandatory rating factors. To the extent that the risks of AVs are transferred to them, the insurance burden passed to consumers in the price of the car can reflect the actual, and presumably lower, risk presented by AVs. As noted above, however, for practical reasons some rating factors, such as annual miles driven and territory, cannot properly be reflected in the automobile price. Moving from the awkward and arbitrary results mandated by Proposition 103’s rating factors to a commercial insurance setting that cannot properly reflect some other rating factors is also an awkward trade-off. At best, it may be a choice of the least worst. Another viable solution might to be to amend the California Insurance Code section 660(a) to exclude from the definition of “policy” those policies covering liability for AVs (at least when operated in autonomous mode). Since Proposition 103 incorporates section 660(a), this would likely require a two-thirds vote of the legislature and the amendment would have to “further the purposes” of Proposition 103. Assuming a two-thirds vote could be mustered, the issue would then be whether the amendment furthers the purposes of the Proposition. To the extent that liability moves from fault-based driving to defect-based products liability, the purposes underlying the mandatory rating factors and the GDD simply cannot be accomplished. Manufacturers will pass these costs through to automobile buyers free of the Proposition’s restraints. Since the purposes of the Proposition, at least with respect to liability coverage,199 simply cannot be accomplished when dealing with self-driving cars, amending section 660(a) would not frustrate the purposes of Proposition 103.

…Filing a “complete rate application with the commissioner” is a substantial impediment to reducing rates. A complete rate application is an expensive, ponderous and time-consuming process. A typical filing may take three to five months before approval. Some applications have even been delayed for a year.205 In 2009, when insurers filed many new rate plans in order to comply with the new territorial rating regulations, delays among the top twenty private passenger auto insurers ranged from a low of 54 days (Viking) to a high of 558 days (USAA and USAA Casualty). Many took over 300 days (e.g., State Farm Mutual, Farmers Insurance Exchange, Progressive Choice).206 …n addition, once an application to lower rates is filed, the Commissioner, consumer groups, and others can intervene and ask that the rates be lowered even further.207 Thus, an application to lower a rate by 6% may invite pressure to lower it even further.208 If they “substantially contributed, as a whole” to the decision, a consumer group can also bill the insurance company for its legal, advocacy, and witness fees.209

…Unless ways can be found to conform Proposition 103 to this new reality, insurance for AVs is likely to migrate to a statutory and regulatory environment untrammeled by Proposition 103-commercial policies carried by manufacturers and suppliers. This migration presents its own set of problems. While the safety of AVs could be more fairly rated, other important rating factors, such as annual miles driven and territory, must be compromised. Whether this migration occurs will also depend on how liability rules do or do not adjust to a world in which people will nevertheless suffer injuries from AVs, but in which it is unlikely our present fault rules will adequately address compensation. If concepts of non-delegable duty, agency, or strict liability attach initial liability to owners of faulty cars with faultless drivers, the insurance burden will first be filtered through automobile insurance governed by Proposition 103. These insurers will then pass the losses up the distribution line to the insurers of suppliers and manufacturers that are not governed by Proposition 103. Manufacturers and suppliers will then pass the insurance cost back to AV owners in the cost of the vehicle. The insurance load reflected in the price of the car will pass through to automobile owners free of any of the restrictions imposed by Proposition 103. There will be no GDD, such as it is, no mandatory rating factors, and, depending on where the suppliers’ or manufacturers’ insurers are located, more flexible rating. One may ask: What is gained by this merry-go-round?

“‘Look Ma, No Hands!’: Wrinkles and Wrecks in the Age of Autonomous Vehicles”, Garza 2012

The benefits of these systems cannot be overestimated given that one-third of drivers admit to having fallen asleep at the wheel within the previous thirty days.31 …If the driver fails to react in time, it applies 40% of the full braking power to reduce the severity of the collision.39 In the most advanced version, the CMBS performs all of the functions described above, and it will also stop the car automatically to avoid a collision when traveling under ten miles-per-hour.40 Car companies are hesitant to push the automatic braking threshold too far out of fear that ‚fully ‘automatic’ braking systems will shift the responsibility of avoiding an accident from the vehicle’s driver to the vehicle’s manufacturer.’41…See Larry Carley, Active Safety Technology: Adaptive Cruise Control, Lane Departure Warning & Collision Mitigation Braking, IMPORT CAR (June 16, 2009), http://www.import-car.com/Article/58867/active_safety_technology_adaptive_cruise_control_lane_departure_warning__collision_mitigation_braking.aspx

…Automobile products liability cases are typically divided into two categories: ‚(1) accidents caused by automotive defects, and (2) aggravated injuries caused by a vehicle’s failure to be sufficiently ‘crashworthy’ to protect its occupants in an accident.‘79 …For example, a car suffers from a design defect when a malfunction in the steering wheel causes a crash. 81 Additionally, plaintiffs have alleged and prevailed on manufacturing- defect claims in cases where ‚unintended, sudden and uncontrollable acceleration’ causes an accident.82 In such cases, plaintiffs have been able to recover under a ‚malfunction theory.’83 Under a malfunction theory, plaintiffs use a ‚res ipsa loquitur like inference to infer defectiveness in strict liability where there was no independent proof of a defect in the product.’84 Plaintiffs have also prevailed where design defects cause injury. 85 For example, there was a proliferation of litigation in the 1970s and 1980s as a result of vehicles that were designed with a high center of gravity, which increased their propensity to roll over.86 Additionally, many design-defect cases arose in response to faulty transmissions that could inadvertently slip into gear, causing crashes and occupants to be run over in some cases. 87 The two primary tests that courts use to assess the defectiveness of a product’s design are the consumer-expectations test and the risk-utility test.88 The consumer-expectations test focuses on whether ‚the danger posed by the design is greater than an ordinary consumer would expect when using the product in an intended or reasonably foreseeable manner.’89 …Thus, while an ordinary consumer can have expectations that a car will not explode at a stoplight or catch fire in a two-mile-per-hour collision, they may not be able to have expectations about how a truck should handle after striking a five- or six-inch rock at thirty-five miles-per-hour.92 Perhaps because the consumer-expectations test is difficult to apply to complex products, and we live in a world where technological growth increases complexity, the risk-utility test has become the dominant test in design-defect cases.93 …Litigation can also arise where a plaintiff alleges that a vehicle is not sufficiently ‚crashworthy.’104 Crashworthiness claims are a type of design- defect claim.105

…Since their advent and incorporation, seat belts have resulted in litigation-much of which has involved crashworthiness claims. 136 In Jackson v. General Motors Corp., for example, the plaintiff alleged that as a result of a defectively designed seat belt, his injuries were enhanced. 137 The defendant manufacturer argued that the complexity of seat belts foreclosed any consumer expectation,138 but the Tennessee Supreme Court noted that seat belts are ‘familiar products for which consumers’ expectations of safety have had an opportunity to develop,‘and permitted the plaintiff to recover under the consumer-expectations test.139 Although manufacturers have been sued where seat belts render a car insufficiently crashworthy- as in cases where they fail to perform as intended or enhance injury-the incorporation of seat belts has reduced liability as well.140 This reduction comes in the form of the ‚seat belt defense.’141 The ’seat belt defense’ allows a defendant to present evidence about an occupant’s nonuse of a seat belt to mitigate damages or to defend against an enhanced-injury claim.142 Because seat belts are capable of reducing the number of lives lost and the overall severity of injuries sustained in crashes, it is argued that nonuse should protect a manufacturer from some claims.143 Although the majority rule is to prevent the admission of such evidence in enhanced-injury litigation, there is a growing trend toward admission.144

…Since their incorporation, consumers have sued manufacturers for defective cruise control systems that lead to injury. 171 Because of the complexity of cruise control technology, courts may not allow a plaintiff to use the consumer-expectations test.172 Despite the complexity of the technology, other courts allow plaintiffs to establish a defect using either the risk-utility test or the consumer-expectations test.173

…Under the consumer-expectations test, manufacturers will likely argue-as they historically have-that OAV technology is too complicated for the average consumer to have appropriate expectations about its capabilities.182 Commentators have stated that ‚consumers may have unrealistic expectations about the capabilities of these technologies . . . . Technologies that are engineered to assist the driver may be overly relied on to replace the need for independent vigilance on the part of the vehicle operator.’183 Plaintiffs will argue that, while the workings of the technology are concededly complex, the overall concept of autonomous driving is not.184 Like the car exploding at a stoplight or the car that catches fire in a two- mile-per-hour collision, the average consumer would expect autonomous vehicles to drive themselves without incident.185 This means that components that are meant to keep the car within a lane will do just that, and others will stop the vehicle at traffic lights. 186 Where incidents occur, OAVs will not have performed as the average consumer would expect.187 …plaintiffs who purchase OAVs at the cusp of availability, and attempt to prove defect under the consumer- expectations test, are likely to face an up-hill battle.194 But the unavailability of the consumer-expectations test will not be a significant detriment as plaintiffs can fall back on the risk-utility test.195 And as OAVs are increasingly incorporated, and users become more familiar with their capabilities, the consumer-expectations test will become more accessible to plaintiffs.196 Given the modern trend, plaintiffs are likely to face the risk- utility test.197

…Additionally, the extent to which injuries are ‚enhanced’ by OAVs will be debated.228 Because the majority of drivers fail to fully apply their brakes prior to a collision,229 where an OAV only partially applies brakes, or fails to apply brakes at all, manufacturers and plaintiffs will disagree about the extent of enhancement.230 Manufacturers will argue that, absent the OAV, the result would have been the same or worse-thus, the extent to which the injuries of the plaintiff are ‚enhanced’ is minimal.231 Plaintiffs will argue that, just like the presentation of crash statistics in a risk-utility analysis, this is a false choice.232 Like no-fire air bag claims, plaintiffs will contend that but for the malfunction of the OAV, their injuries would have been greatly reduced or nonexistent. 233 As a result, any injuries sustained above that threshold should serve as a basis for recovery. 234

…In products liability cases the ’use of expert witnesses has grown in both importance and expense.’301 Because of the extraordinary cost of experts in products liability litigation, many plaintiffs are turned away because, even if they were to recover, the prospective award would not cover the expense of litigating the claim. 302

…Although complex, OAVs function much like the cruise control that exists in modern cars. As we have seen with seat belts, air bags, and cruise control, manufacturers have always been hesitant to adopt safety technologies. Despite concerns, products liability law is capable of handling OAVs just as it has these past technologies. While the novelty and complexity of OAVs are likely to preclude plaintiffs from proving defect under the consumer-expectation test, as implementation increases this likelihood may decrease. Under a risk-utility analysis, manufacturers will stress the extraordinary safety benefits of OAVs, while consumers will allege that designs can be improved. In the end, OAV adoption will benefit manufacturers. Although liability will fall on manufacturers when vehicles fail, decreased incidences and severity of crashes will result in a net decrease in liability. Further, the combination of LDWS cameras and EDRs will drastically reduce the cost of litigation. By reducing reliance on experts for complex causation determinations, both manufacturers and plaintiffs will benefit. In the end, obstacles to OAV implementation are more likely to be psychological than legal, and the sooner that courts, manufacturers, and the motoring public prepare to confront these issues, the sooner lives can be saved.

“Self-driving cars can navigate the road, but can they navigate the law? Google’s lobbying hard for its self-driving technology, but some features may never be legal”, The Verge 14 December 2012

Google says that on a given day, they have a dozen autonomous cars on the road. This August, they passed 300,000 driver-hours. In Spain this summer, Volvo drove a convoy of three cars through 200 kilometers of desert highway with just one driver and a police escort.

…Bryant Walker Smith teaches a class on autonomous vehicles at Stanford Law School. At a workshop this summer, he put forward this thought experiment: the year is 2020, and a number of companies offer “advanced driver assistance systems” with their high-end model. Over 100,000 units have been sold. The owner’s manual states that the driver must remain alert at all times, but one night a driver - we’ll call him “Paul” - falls asleep while driving over a foggy bridge. The car tries to rouse him with alarms and vibrations but he’s a deep sleeper, so the car turns on the hazard lights and pulls over to the side of the road where another driver (let’s say Julie) rear-ends him. He’s injured, angry, and prone to litigation. So is Julie. That would be tricky enough by itself, but then Smith starts layering on complications. Another model of auto-driver would have driven to the end of the bridge before pulling over. If Paul had updated his software, it would have braced his seatbelt for the crash, mitigating his injuries, but he didn’t. The company could have pushed the update automatically, but management chose not to. Now, Smith asks the workshop, who gets sued? Or for a shorter list, who doesn’t?

…The financial stakes are high. According to the Insurance Research Council, auto liability claims paid out roughly $215 for each insured car, between bodily injury and property damage claims. With 250 million cars on the road, that’s $54 billion a year in liability. If even a tiny portion of those lawsuits are directed towards technologists, the business would become unprofitable fast.

…Changing the laws in Europe would take a replay of the internationally ratified Vienna Convention (passed in 1968) as well as pushing through a hodgepodge of national and regional laws. As Google proved, it’s not impossible, but it leaves SARTRE facing an unusually tricky adoption problem. Lawmakers won’t care about the project unless they think consumers really want it, but it’s hard to get consumers excited about a product that doesn’t exist yet. Projects like this usually rely on a core of early adopters to demonstrate their usefulness - a hard enough task, as most startups can tell you - but in this case, SARTRE has to bring auto regulators along for the ride. Optimistically, Volvo told us they expect the technology to be ready “towards the end of this decade,” but that may depend entirely on how quickly the law moves. The less optimistic prediction is that it never arrives at all. Steve Shladover is the program manager of mobility at California’s PATH program, where they’ve been trying to make convoy technology happen for 25 years, lured by the prospect of fitting three times as many cars on the freeway. They were showing off a working version as early as 1997 (powered by a single Pentium processor), before falling into the same gap between prototype and final product. “It’s a solvable problem once people can see the benefits,” he told The Verge, “but I think a lot of the current activity is wildly optimistic in terms of what can be achieved.” When I asked him when we’d see a self-driving car, Shladover told me what he says at the many auto conferences he’s been to: “I don’t expect to see the fully-automated, autonomous vehicle out on the road in the lifetime of anyone in this room.”

…Many of Google’s planned features may simply never be legal. One difficult feature is the “come pick me up” button that Larry Page has pushed as a solution to parking congestion. Instead of wasting energy and space on urban parking lots, why not have cars drop us off and then drive themselves to park somewhere more remote, like an automated valet?It’s a genuinely good idea, and one Google seems passionate about, but it’s extremely difficult to square with most vehicle codes. The Geneva Convention on Road Traffic (1949) requires that drivers “shall at all times be able to control their vehicles,” and provisions against reckless driving usually require “the conscious and intentional operation of a motor vehicle.” Some of that is simple semantics, but other concerns are harder to dismiss. After a crash, drivers are legally obligated to stop and help the injured - a difficult task if there’s no one in the car. As a result, most experts predict drivers will be legally required to have a person in the car at all times, ready to take over if the automatic system fails. If they’re right, the self-parking car may never be legal.

“Automated Vehicles are Probably Legal in the United States”, Bryant Walker Smith 2012

The short answer is that the computer direction of a motor vehicle’s steering, braking, and accelerating without real-time human input is probably legal….The paper’s largely descriptive analysis, which begins with the principle that everything is permitted unless prohibited, covers three key legal regimes: the 1949 Geneva Convention on Road Traffic, regulations enacted by the National Highway Traffic Safety Administration (NHTSA), and the vehicle codes of all fifty US states.

The Geneva Convention, to which the United States is a party, probably does not prohibit automated driving. The treaty promotes road safety by establishing uniform rules, one of which requires every vehicle or combination thereof to have a driver who is “at all times … able to control” it. However, this requirement is likely satisfied if a human is able to intervene in the automated vehicle’s operation.

NHTSA’s regulations, which include the Federal Motor Vehicle Safety Standards to which new vehicles must be certified, do not generally prohibit or uniquely burden automated vehicles, with the possible exception of one rule regarding emergency flashers. State vehicle codes probably do not prohibit-but may complicate-automated driving. These codes assume the presence of licensed human drivers who are able to exercise human judgment, and particular rules may functionally require that presence. New York somewhat uniquely directs a driver to keep one hand on the wheel at all times. In addition, far more common rules mandating reasonable, prudent, practicable, and safe driving have uncertain application to automated vehicles and their users. Following distance requirements may also restrict the lawful operation of tightly spaced vehicle platoons. Many of these issues arise even in the three states that expressly regulate automated vehicles.

…This paper does not consider how the rules of tort could or should apply to automated vehicles-that is, the extent to which tort liability might shift upstream to companies responsible for the design, manufacture, sale, operation, or provision of data or other services to an automated vehicle. 6

…Because of the broad way in which the term and others like it are defined, an automated vehicle probably has a human “driver.” 295 Obligations imposed on that person may limit the independence with which the vehicle may lawfully operate. 296 In addition, the automated vehicle itself must meet numerous requirements, some of which may also complicate its operation. 297 Although three states have expressly established the legality of automated vehicles under certain conditions, their respective laws do not resolve many of the questions raised in this section. 298

…A brief but important aside: To varying degrees, states impose criminal or quasicriminal liability on owners who permit others to drive their vehicles. 359 In Washington, “[b]oth a person operating a vehicle with the express or implied permission of the owner and the owner of the vehicle are responsible for any act or omission that is declared unlawful in this chapter. The primary responsibility is the owner’s.” 360 Some states permit an inference that the owner of a vehicle was its operator for certain offenses; 361 Wisconsin provides what is by far the most detailed statutory set of rebuttable presumptions. 362 Many others punish owners who knowingly permit their vehicles to be driven unlawfully. 363 Although these owners are not drivers, they are assumed to exercise some judgment or control with respect to those drivers-an instance of vicarious liability that suggests an owner of an automated vehicle might be liable for merely permitting its automated operation. 364

…On the human side, physical presence would likely continue to provide a proxy for or presumption of driving. 366 In other words, an individual who is physically positioned to provide real-time input to a motor vehicle may well be treated as its driver. This is particularly likely at levels of automation that involve human input for certain portions of a trip. In addition, an individual who starts or dispatches an automated vehicle, who initiates the automated operation of that vehicle, or who specifies certain parameters of operation probably qualifies as a driver under existing law. That individual may use some device-anything from a physical key to the click of a mouse to the sound of her voice-to activate the vehicle by herself. She may likewise deliberately request that the vehicle assume the active driving task. And she may set the vehicle’s maximum speed or level of assertiveness. This working definition is unclear in the same ways that existing law is likely to be unclear. Relevant acts might occur at any level of the primary driving task, from a decision to take a particular trip to a decision to exceed any speed limit by ten miles per hour. 367 A tactical decision like speeding is closely connected with the consequences-whether a moving violation or an injury-that may result. But treating an individual who dispatches her fully automated vehicle as the driver for the entirety of the trip could attenuate the relationship between legal responsibility and legal fault. 368 Nonetheless, strict liability of this sort is accepted within tort law 369 and present, however controversially, in US criminal law. 370

On the corporate side, a firm that designs or supplies a vehicle’s automated functionality or that provides data or other digital services might qualify as a driver under existing law. The key element, as provided in the working definition, may be the lack of a human intermediary: A human who provides some input may still seem a better fit for a human-centered vehicle code than a company with other relevant legal exposure. However, as noted above, public outrage is another element that may motivate new uses of existing laws. 377

…The mechanism by which someone other than a human would obtain a driving license is unclear. For example, some companies may possess great vision, but “a test of the applicant’s eyesight” may nonetheless be difficult. 395 And while General Motors may (or may not) 396 meet a state’s minimum age requirement, Google would not. [See Google, Google’s mission is to organize the world’s information and make it universally accessible and useful, www.google.com/intl/en/about/company/. In some states, Google might be allowed to drive itself to school. See, e.g., Nev. Rev. Stat. § 483.270; Nev. Admin. Code § 483.200.]

And people say lawyers have no sense of humor.

CC Licenced

[Link] Philosophical Parenthood

1 SquirrelInHell 30 May 2017 02:09PM

Comments (18)

CC Licenced

Divergent preferences and meta-preferences

4 Stuart_Armstrong 30 May 2017 07:33AM

Crossposted at the Intelligent Agents Forum.

In simple graphical form, here is the problem of divergent human preferences:

Here the AI either chooses A or ¬A, and as a consequence, the human then chooses B or ¬B.

There are a variety of situations in which this is or isn't a problem (when A or B or their negations aren't defined, take them to be the negative of what is define):

Not problems:
- A/¬A = "gives right shoe/left shoe", B/¬B = "adds left shoe/right shoe".
- A = "offers drink", ¬B = "goes looking for extra drink".
- A = "gives money", B = "makes large purchase".
Potentially problems:
- A/¬A = "causes human to fall in love with X/Y", B/¬B = "moves to X's/Y's country".
- A/¬A = "recommends studying X/Y", B/¬B = "choose profession P/Q".
- A = "lets human conceive child", ¬B = "keeps up previous hobbies and friendships".
Problems:
- A = "coercive brain surgery", B = anything.
- A = "extreme manipulation", B = almost anything.
- A = "heroin injection", B = "wants more heroin".

So, what are the differences? For the "not problems", it makes sense to model the human as having a single reward R, variously "likes having a matching pair of shoes", "needs a certain amount of fluids", and "values certain purchases". Then all that the the AI is doing is helping (or not) the human towards that goal.

As you move more towards the "problems", notice that they seem to have two distinct human reward functions, R_A and R_¬A, and that the AI's actions seem to choose which one the human will end up with. In the spirit of humans not being agents, this seems to be AI determining what values the human will come to possess.

Grue, Bleen, and agency

Of course, you could always say that the human actually has reward R = I_AR_A + (1-I_A)R_¬A, where I_A is the indicator function as to whether the AI does action A or not.

Similarly to the grue and bleen problem, there is no logical way of distinguishing that "pieced-together" R from a more "natural" R (such as valuing pleasure, for instance). Thus there is no logical way of distinguishing the human being an agent from the human not being an agent, just from its preferences and behaviour.

However, from a learning and computational complexity point of view, it does make sense to distinguish "natural" R's (where R_A and R_¬A are essentially the same, despite the human's actions being different) from composite R's.

This allows us to define:

Preference divergence point: A preference divergence point is one where R_A and R_¬A are sufficiently distinct, according to some criteria of distinction.

Note that sometimes, R_A = R_A' + R' and R_¬A = R_¬A' + R': the two R_A and R_¬A overlap on a common piece R', but diverge on R_A' and R_¬A'. It makes sense to define this as a preference divergence point as well, if R_A'and R_¬A' are "important" in the agent's subsequent decisions. Importance being a somewhat hazy metric, which would, for instance, assess how much R' reward the human would sacrifice to increase R_A' and R_¬A'.

Meta-preferences

From the perspective of revealed preferences about the human, R(μ)=I_AR_A + μ(1-I_A) R_¬A will predict the same behaviour for all scaling factors μ > 0.

Thus at a preference divergence point, the AI's behaviour, if it was a R(μ) maximiser, would depend on the non-observed weighting between the two divergent preferences.

This is unsafe, especially if one of the divergent preferences is much easier to achieve a high value with than the other.

Thus preference divergence points are moments when the AI should turn explicitly to human meta-preferences to distinguish between them.

This can be made recursive - if we see the human meta-preferences as explicitly weighting R_A versus R_¬A and hence giving R, then if there is a prior AI decision point Z, and, depending on what the AI chooses, the human meta-preferences will be different, this gives two reward functions R_Z=I_AR_A+ μ_Z(1-I_A)R_¬A and R_¬Z=I_AR_A+ μ_¬Z(1-I_A)R_¬A with different weights μ_Z and μ_¬Z.

If these weights are sufficiently distinct, this could identify a meta-preference divergence point and hence a point where human meta-meta-preferences become relevant.

Comments (1)

CC Licenced

[Link] "AIXIjs: A Software Demo for General Reinforcement Learning", Aslanides 2017

0 gwern 29 May 2017 09:09PM

Comments (1)

CC Licenced

Invitation to comment on a draft on multiverse-wide cooperation via alternatives to causal decision theory (FDT/UDT/EDT/...)

2 Caspar42 29 May 2017 08:34AM

I have written a paper about “multiverse-wide cooperation via correlated decision-making” and would like to find a few more people who’d be interested in giving a last round of comments before publication. The basic idea of the paper is described in a talk you can find here. The paper elaborates on many of the ideas and contains a lot of additional material. While the talk assumes a lot of prior knowledge, the paper is meant to be a bit more accessible. So, don’t be disheartened if you find the talk hard to follow — one goal of getting feedback is to find out which parts of the paper could be made more easy to understand.

If you’re interested, please comment or send me a PM. If you do, I will send you a link to a Google Doc with the paper once I'm done with editing, i.e. in about one week. (I’m afraid you’ll need a Google Account to read and comment.) I plan to start typesetting the paper in LaTeX in about a month, so you’ll have three weeks to comment. Since the paper is long, it’s totally fine if you don’t read the whole thing or just browse around a bit.

CC Licenced

Open thread, May 29 - June 4, 2017

1 Thomas 29 May 2017 06:13AM

If it's worth saying, but not worth its own post, then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should start on Monday, and end on Sunday.

4. Unflag the two options "Notify me of new top level comments on this article" and "Make this post available under..." before submitting

Comments (61)

[Link] Interview on IQ, genes, and genetic engineering with expert (Hsu)

4 James_Miller 28 May 2017 10:19PM

Comments (5)

CC Licenced

Bi-Weekly Rational Feed

13 deluks917 28 May 2017 05:12PM

Five Recommended Articles You Might Have Missed:

The Four Blind Men The Elephant And Alan Kay by Meredith Paterson (Status 451) - Managing technical teams. Taking a new perspective is worth 90 IQ points. Getting better enemies. Guerrilla action.

Vast Empirical Literature by Marginal REVOLUTION - Tyler's 10 thoughts on approaching fields with large literatures. He is critical of Noah's "two paper rule" and recommends alot of reading.

Notes From The Hufflepuff Unconference (Part 1) by Raemon (lesswrong) - Goal: Improve at: "social skills, empathy, and working together, sticking with things that need sticking with". The article is a detailed breakdown of the unconference including: Ray's Introductory Speech, a long list of what people want to improve on, the lightning talks, the 4 breakout sessions, proposed solutions, further plans, and closing words. Links to conference notes are included for many sections.

Antipsychotics Might Cause Cognitive Impairment by Sarah Constantin (Otium) - A harrowing personal account of losing abstract thinking ability on Risperdal. The author conducts a literature review, and concludes with some personal advice about taking medication.

Dwelling In Possibility by Sarah Constantin (Otium) - Leadership. Confidence in the face of the uncertainty and imperfection. Losing yourself when you try to step back and facilitate.

Scott:

Those Modern Pathologies by Scott Alexander - You can argue X is a modern pathology for almost any value of X. Scott demonstrates this by repeated example. Among other things "Aristotelian theory of virtue" and "Homer's Odyssey" get pathologized.

The Atomic Bomb Considered As Hungarian High School Science Fair Project by Scott Alexander - Ashkenazi Jewish Intelligence. An explanation of Hungarian dominance in physics and science in the mid 1900s.

Classified Ads Thread by Scott Alexander - Open thread where people post ads. People are promoting their websites and some of them are posting actual job ads among other things.

Open Thread 76 by Scott Alexander - Bi-weekly Open thread.

Postmarketing Surveillance Is Good And Normal by Scott Alexander - Scott shows why a recent Scientific American study does not imply the FDA is too risky.

Epilogue by Scott Alexander (Unsong) - All's Whale that Ends Whale.

Polyamory Is Not Polygyny by Scott Alexander - A quick review of how polyamory actually function in the rationalist community.

Bail Out by Scott Alexander - "About a fifth of the incarcerated population – the top of the orange slice, in this graph – are listed as “not convicted”. These are mostly people who haven’t gotten bail. Some are too much of a risk. But about 40% just can’t afford to pay."

Rationalist:

Strong Men Are Socialist Reports A Study That Previously Reported The Opposite by Jacob Falkovich (Put A Number On It!) - Defense Against the Dark Statistical Arts. Jacob provides detailed commentary on a popular study and shows that the studies dataset can be used to support the opposite conclusion, with p = 0.0086.

Highly Advanced Tulpamancy 101 For Beginners by H i v e w i r e d - Application of lesswrong theory to the concept of the self. In particular the author applies "How an Algorithm Feels from the Inside" and "Map and Territory". Hive then goes into the details of creating and interacting with tulpas. "A tulpa is an autonomous entity existing within the brain of a “host”. They are distinct from the host in that they possess their own personality, opinions, and actions"

Existential Risk From Ai Without An Intelligence by Alex Mennen (lesswrong) - Reasons why an intelligence explosion might not occur and reasons why we might have a problem anyway.

Dragon Army Theory Charter (30min Read) by Duncan Sabien (lesswrong) - A detailed plan for an ambitious military style rationalist house. The major goals include self-improvement, high quality group projects and the creation of a group with absolute trust in one another. The leader of the house is the curriculum director and head of product at CFAR.

The Story Of Our Life by H i v e w i r e d - The authors explain their pre-rationalist life and connection to the community. They then argue the rationalist community should take better care of one another. "Venture Rationalism".

Don't Believe in God by Tyler Cowen - Seven arguments for not believing in God. Among them: Lack of Bayesianism among believers, the degree to which people follow their family religion and the fundamental weirdness of reality.

The Four Blind Men The Elephant And Alan Kay by Meredith Paterson (Status 451) - Managing technical teams. Taking a new perspective is worth 90 IQ points. Getting better enemies. Guerrilla action.

Qualia Computing At Consciousness Hacking June 7th 2017 by Qualia Computing - Qualia computing will present in San Fransisco on June 7th at Consciousness Hacking. The event description is detailed and should give readers a good intro to Qualia Computing's goals. The author's research goal is to create a mathematical theory of pain/pleasure and be able to measure these directly from brain data.

Is Silicon Valley Real by Ben Hoffman (Compass Rose) - The old culture of Silicon Valley is mostly gone, replaced by something overpriced and materialist. Ben check's the details of Scott Alexander's list of six noble startups and finds only two in SV proper.

Why Is Harry Potter So Popular by Ozy (Thing of Things) - Ozy discusses a paper on song popularity in an artificial music market. Social dynamics had a big impact on song ratings. "Normal popularity is easily explicable by quality. Stupid, wild, amazing popularity is due to luck."

Design A Better Chess by Robin Hanson - Can we design a game that promotes even more useful honesty than chess? A link to Hanson's review of Gary Kasparov's book is included.

Deserving Truth 2 by Andrew Critch - How the author's values changed over time. Originally he tried to maximize his own positive sensory experiences. The things he cared about began to include more things, starting with his GF's experiences and values. He eventually rejects "homo-economus" thinking.

A Theory Of Hypocrisy by João Eira (Lettuce be Cereal) - Hypocrisy evolved as a way to solve free rider problems. "It pays to be a free rider. If no one finds out"

Building Community Institution In Five Hours a Week by Particular Virtue - Eight pieces of advice for running a successful meetup. The author and zir partner have been running lesswrong events for five years.

Dwelling In Possibility by Sarah Constantin (Otium) - Leadership. Confidence in the face of the uncertainty and imperfection. Losing yourself when you try to step back and facilitate.

Ai Safety Three Human Problems And One Ai Issue by Stuart Armstrong (lesswrong) - Humans have poor predictions, don't know their values and aren't agents. Ai might be very powerful. A graph of which problems many Ai risk solutions target.

Recovering From Failure by mindlevelup - Avoid negative spirals, figure out why you failed, List of questions to ask yourself. Strategies -> Generate good alternatives, metacognitive affordances.

Review The Dueling Neurosurgeons by Sam Kean by Aceso Under Glass - Positive review. Author learned alot. Speculation on a better way to teach Science.

Principia Qualia Part 2: Valence by Qualia Computing - A mathematical theory of valence (what makes experience feel good or bad). Speculative but the authors make concrete predictions. Music plays a heavy role.

Im Not Seaing It by Robin Hanson - Arguments against seasteading.

EA:

One of the more positive surprises by GiveDirectly - Links post. Eight articles on Give Directly, Cash Transfer and Basic Income.

Returns Functions And Funding Gaps by the Center for Effective Altruism (EA forum) - Links to CEA's explanation of what "returns functions" are and how using them compares to "funding gap" model. They give some arguments why returns functions are a superior model.

Online Google Hangout On Approaches To by whpearson (lesswrong) - Community meeting to discuss Ai risk. Will use "Optimal Brainstorming Theory". Currently early stage. Sign up and vote on what times you are available.

Expected Value Estimates We Cautiously Took by The Oxford Prioritization Project (EA forum) - Details of how the four bayesian probability models were compared to produce a final decision. Some discussion of how assumptions affect the final result. Actual code is included.

Four Quantitative Models Aggregation And Final by The Oxford Prioritization Project (EA forum) - 80K hours, MIRI, Good Foods Institute and StrongMinds were considered. Decisions were made using concrete Bayesian EV calculations. Links to the four models are included.

Peer to Peer Aid: Cash in the News by GiveDirectly - 8 Links about GiveDirectly, cash transfer and basic income.

The Value Of Money Going To Different Groups by The Center for Effective Altruism - "It is well known that an extra dollar is worth less when you have more money. This paper describes the way economists typically model that effect, using that to compare the effectiveness of different interventions. It takes remittances as a particular case study."

Politics and Economics:

Study Of The Week Better And Worse Ways To Attack Entrance Exams by Freddie deBoer - Freddie's description of four forms of "test validity". The SAT and ACT are predictive of college grades, one should criticize them from other angles. Freddie briefly gives his socialist critique.

How To Destroy Civilization by Zvi Moshowitz - A parable about the game "Advanced Civilization". The difficulties of building a coalition to lock out bad actor. Donald Trump. [Extremely Partisan]

Trust Assimilation by Bryan Caplan - Data on how much immigrants and their children trust other people. How predictive is the trust level of their ancestral country. Caplan reviews papers and crunches the numbers himself.

There Are Bots, Look Around by Renee DiResta (ribbonfarm) - High frequency trading disrupted finance. Now algorithms and bots are disrupting the marketplace of ideas. What can finance's past teach us about politics' future?

The Behavioral Economics of Paperwork by Bryan Caplan - Vast Numbers of students miss financial aid because they don't fill out paperwork. Caplan explores the economic implications of the fact that "Humans hate filling out paperwork. As a result, objectively small paperwork costs plausibly have huge behavioral response".

The Nimby Challenge by Noah Smith - Smith Argues makes an economic counterargument to the claims that building more housing wouldn't lower prices. Noah includes 6 lessons for engaging with NIMBYs.

Study Of The Week What Actually Helps Poor Students: Human Beings by Freddie deBoer - Personal feedback, tutoring and small group instruction had the largest positive effect. Includes Freddie's explanation of meta-analysis.

Vast Empirical Literature by Marginal REVOLUTION - Tyler's 10 thoughts on approaching fields with large literatures. He is critical of Noah's "two paper rule" and recommends alot of reading.

Impact Housing Price Restrictions by Marginal REVOLUTION - Link to a job market paper on the economic effects of housing regulation.

Me On Anarcho Capitalism by Bryan Caplan - Bryan is interviewed on the Rubin Report about Ancap.

Campbells Law And The Inevitability Of School Fraud by Freddie deBoer - Rampant Grade Inflation. Lowered standards. Campbell's law says that once you base policy on a metric that metric will always start being gamed

Nimbys Economic Theories: Sorry Not Sorry by Phil (Gelman's Blog) - Gelman got a huge amount of criticism on his post on whether building more housing will lower prices in the Bay. He responds to some of the criticism here. Long for Gelman.

Links 8 by Artir (Nintil) - Link Post. Physics, Technology, Philosophy, Economics, Psychology and Misc.

Arguing About How The World Should Burn by Sonya Mann ribbonfarm - Two different ways to decide who to exclude. One focuses on process the other on content. Scott Alexander and Nate Soares are quoted. Heavily [Culture War].

Seeing Like A State by Bayesian Investor - A quick review of "Seeing like a state".

Whats Up With Minimum Wage by Sarah Constantin (Otium) - A quick review of the literature on the minimum wage. Some possible explanations for why raising it not reduce unemployment.

Misc:

Entirely Too Many Pieces Of Unsolicited Advice To Young Writer Types by Feddie deBoer - Advice about not working for free, getting paid, interacting with editors, why 'Strunk and White' is awful, and taking writing seriously.

Conversations On Consciousness by H i v e w i r e d - The author is a plural system. Their hope is to introduce plurality by doing the following: "First, we’re each going to describe our own personal experiences, from our own perspectives, and then we’re going to discuss where we might find ourselves within the larger narrative regarding consciousness."

Notes On Debugging Clojure Code by Eli Bendersky - Dealing with Clojure's cryptic exceptions, Finding which form an exception comes from, Trails and Logging, Deeper tracing inside cond forms

How to Think Scientifically About Scientists’ Proposals for Fixing Science by Andrew Gelman - Gelman asks how to scientifically evaluate proposals to fix science. He considers educational, statistical, research practice and institutional reforms. Excerpts from an article Gelman wrote, the full paper is linked.

Call for Volunteers who Want to Exercize by Aceso Under Glass - Author is looking for volunteers who want to treat their anxiety or mood disorder with exercise.

Learning Deep Learning the Easy Way with Keras (lesswrong) - Articles showing the power of neural networks. Discussion of ML frameworks. Resources for learning.

Unsong of Unsongs by Scott Aaronson - Aaronson went to the Unsong wrap party. A quick review of Unsong. Aaronson talks about how Scott Alexander defended him with untitled.

2016 Spending by Mr. Money Mustache - Full details of last year's budget. Spending broken down by category.

Amusement:

And Another Physics Problem by protokol2020 - Two Planets. Which has a higher average surface temperature.

A mysterious jogger by Jacob Falkovich (Put A Number On It!) - A mysterious jogger. Very short fiction.

Podcast:

Persuasion And Control by Waking Up with Sam Harris - "surveillance capitalism, the Trump campaign's use of Facebook, AI-enabled marketing, the health of the press, Wikileaks, ransomware attacks, and other topics."

Raj Chetty: Inequality, Mobility and the American Dream by Conversations with Tyler - "As far as I can tell, this is the only coverage of Chetty that covers his entire life and career, including his upbringing, his early life, and the evolution of his career, not to mention his taste in music"

Is Trump's incompetence saving us from his illiberalism? by The Ezra Klein Show - Political Scientist Yascha Mounk. "What Mounk found is that the consensus we thought existed on behalf of democracy and democratic norms is weakening."

The Moral Complexity Of Genetics by Waking Up with Sam Harris - "Sam talks with Siddhartha Mukherjee about the human desire to understand and manipulate heredity, the genius of Gregor Mendel, the ethics of altering our genes, the future of genetic medicine, patent issues in genetic research, controversies about race and intelligence, and other topics."

Ester Perel by The Tim Ferriss - The Relationship Episode: Sex, Love, Polyamory, Marriage, and More

Lane Pritchett by Econtalk - Growth, and Experiments

Meta Learning by Tim Ferriss - Education, accelerated learning, and my mentors. Conversation with Charles Best the founder and CEO of DonorsChoose.org

Bryan Stevenson On Why The Opposite Of Poverty Isn't Wealth by The Ezra Klein Show - Founder and executive director of the Equal Justice Initiative. Justice for the wrongly convicted on Death Row.

Therapists have a financial incentive to re-traumatise patients.

CC Licenced

10 'incredible' weaknesses of the mental health system

7 arunbharatula 28 May 2017 04:22AM

I aim to identify some of the mental health workforce's credibility issues in this article. This may inform your prevention and treatment strategy as a mental health consumer, or your practice if you work in mental health.

Mental health is the strongest determinant of quality of life at a later age. And, the pursuit of happiness predicts both positive emotions and less depressive symptoms. People who prioritize happiness are more psychologically able. In times of crises, some turn to the mental health system for support. But, how credible is the support available? Here are 10 categories of shortcomings that the mental health sector faces today:

1. Institutional credibility

Headspace's evaluations indicate it’s ineffective and they are evaluated better than many services out there. This isn’t academic, attendees who report that their mental health has not improved since using the service will trust the mental health system less, and with good reason.

2. Network credibility

There is an evidence base for the selecting a type of therapy (psychodynamic, cognitive-behavioural, etc) for a particular constellations of mental symptoms. If you work in mental health, have you ever made a referral on the basis of both symptomatology and theoretical orientation?

3. ‘Walk the talk’ credibility

Social workers, nurses, social workers medical doctors, and psychiatrists abuse substances and incur mental ill-health at among the highest rates of any occupation. For instance, the psychiatrist burnout rate is 40%. Mental health consumers may perceive clinicians as hypocritical or unwilling (...or too willing) to swallow their own medicine.

4. Academic credibility

Psychology is mired by error-riddled research and myth-ridden textbooks. Broadly, most published research is wrong. And, questionable research practices are common which bias the relevant evidence.

The difference between a well designed experiment and a poorly designed psychotherapy experiment is large. To quote the pseudonymous physician Scott Alexander:

‘Low-quality psychotherapy trials in general had a higher effect size (SMD = 0.74) than high-quality trials (SMD = 0.22), p < 0.001"...Effect sizes for the low quality trials are triple those for the high-quality trials.’

5. Credibility of treatments

Are treatments are becoming less effective over time? Cognitive behavioural therapy is a common treatment for various mental illnesses. It is the most researched psychotherapy. However, the more evidence piles up, the less effective that psychotherapy appears to be...the same goes for antidepressants.

Why are outdated treatments still used? Over the 19th and 20th Centuries, Austrian neurologist Sigmund Freud famously founded ‘psychoanalysis’. Psychoanalysis is a school of psychotherapy that together with other 'psychodynamic' psychotherapies focused on early experience on human behaviour and emotion. Freud's ideas challenged fundamental assumptions about human psychology. In particular, he suggested that our conscious mind is the just the tip of iceberg of our identities.

Today Freud is the subject of jokes and derision. Many of his testable ideas have been proven false. 'When tested, psychoanalysis was shown to be less effective than placebo.’ Yet, many psychologists and psychiatrists continue to practice psychoanalysis.

Psychology is a rather unsettled science. One estimate for the time after which half of the ‘knowledge’ in the field of psychology is overturned or superseded (it’s ‘half-life’) is at just 7.5 years. Interestingly, this time-span appears to be falling. That would suggest the field is becoming increasingly less reliable. The subfield of psychoanalysis bucks the trend. It has over double the parent field’s half-life. Why?

How do other subfields of psychology fair? Psychopharmacology is at the intersection of psychiatric drugs and brain chemistry. Knowledge in psychopharmacology is overturned at a rate higher than the rest of the field in general. Typically the ‘half life of knowledge’ argument aims discount psychology relative to ‘harder’ sciences like physics.

Psychological therapies are confusing and unnecessarily fragmented: According to The Handbook of Counseling Psychology:

‘Meta-analyses of psychotherapy studies have consistently demonstrated that there are no substantial differences in outcomes among treatments.’

Meta-analyses are a kind of research technique that quantitatively puts together many pieces of individual relevant research on a particular topic. There is 'little evidence to suggest that any one psychological therapy consistently outperforms any other for any specific psychological disorders.

This is sometimes called the 'Dodo bird verdict' after a scene/section in Alice in Wonderland where every competitor in a race was called a winner and is given prizes'. So, what is one to make of the best vetted clinical guidelines that indicate that particular therapies are more appropriate for particular mental conditions?

Guidelines are considered a higher order of evidence than a ‘handbook’ to some, and vice-versa for another. Could an expert or indeed an amateur credibly lead someone to conclude that all therapies are ‘equal’ or ‘different’ armed with either body of evidence? Could a similar case be made for say, antibiotics? Yes, or so the evidence suggests in the case of antibiotics, actually.

Finally, psychological therapies are administered haphazardly. Eclectically combining elements from different psychological therapies is inefficient. But, it happens. Clinicians should ‘integrate’ components of different psychotherapies using established formulae, if they want to ‘mix and match’. When I hear someone’s theoretical orientation is ‘psychodynamically informed’ or similar, for me that’s a red flag for eccelectisms.

6. Economic credibility

7. Social credibility

'The benefits of psychotherapy may be no better than the benefits of talking to a friend'.

8. Credibility of counsel

Mental health professionals offer their clients and the community general counsel and advice. But, if I was to ask a given mental health professional about the value of kindness or love of learning they would almost certainly indicate it’s worthwhile. Pop psychology is pervasive. And why not, people have been interested in psychology long before it was a science. But, misconceptions about psychology infiltrate mental health care practice.

Researchers who have reported on the character traits of people with high and low life satisfaction found something like this:

Character strengths that DO predict life satisfaction	Character strengths that DO NOT predict life satisfaction
Zest	Appreciation of beauty and excellence
Curiosity	Creativity
Hope	kindness
Humour	Love of learning
	Perspective

Meanwhile, research that separates their findings by gender looks different

Character strengths that predict life satisfaction

Men	Women
humour	zest
fairness	gratitude
perspective	hope
creativity	appreciation of beauty and love

Would you receive nuanced, evidence-based advice when soliciting general counsel from your treatment provider?

9. Practitioner credibility

Consider the therapist factors that relate to a patient's success in therapy:

What does predict success?	What there aren’t stable conclusions about
Compliance with a treatment manual (but that compromises a therapist’s relationship skills and supportiveness)	Interpersonal style of therapist
Female therapists	Verbal style of therapist
Ethnic similarity of therapist and patient	Nonverbal styles of therapist
Ethnic sensitivity of therapist to patient	Combined verbal and nonverbal patterns
Therapists with more training	Which treatment manual is used
	Therapist disclosure about themselves
	Therapist directness
	Therapist interpretation of their relationship with the patient, their motives and their psychological processes
	Therapist personality
	Therapist coping patterns
	Therapist emotional wellbeing
	Therapist values
	Therapist beliefs
	Therapists cultural beliefs
	Therapist dominance
	Therapist sense of control
	Therapist sense of what a patient's needs to know

Are mental health services hiring based on the factors that predict a consumer’s success in therapy? Are they training for the right skills, and ignoring those that are irrelevant?

10. Diagnostic credibility

Imprecise measurement and lack of gold standards for validating diagnoses means that definitions tend to drift over time, even though, per the evidence, response to treatment does not vary across culture.

45% of Australians will experience mental illness over their lifetime. Whether that mental ill-health is transient, long-term or lifelong matters to the individual and for public health. To illustrate: experts suggests that those who have had 2 depressive episodes in recent years, or three episodes over their lifelong to get treated on an ongoing basis to prevent recurrent depression.

'At least 60% of individuals who have had one depressive episode will have another, 70% of individuals who have had two depressive episodes will have a third, and 90% of individuals with three episodes will have a fourth episode. '

- APA

Without reliable diagnoses, how can one estimate their risk of relapse into depression?

Comments (6)

CC Licenced

[Link] Researchers studying century-old drug in potential new approach to autism

0 morganism 27 May 2017 09:16PM

On "Overthinking" Concepts

3 Bound_up 27 May 2017 05:07PM

I've NOT been confused by the problem of overthinking in the middle of performing an action. I understand perfectly well the disadvantages of using system 2 in a situation where time is sufficiently limited.

And maybe there are some other fail modes where overthinking has some disadvantages.

But there's one situation where I'd often be accused by someone of "overthinking" something when I didn't even understand what they might mean, and that was in understanding concepts. I would think "Huh? How can thinking less about the concept you're explaining help me understand that concept more? I don't currently understand it; I can't just stay here! Even if you thought I needed to take longer to try and understand this, or that I needed more experience or to shorten the inferential gap, all of that would mean doing more thinking, not less."

Then I would think "Well, I must be misunderstanding the way they're using the word 'overthinking,' that's all." I'd ask for a clear explanation and...

"You're overthinking it."

Now I was overthinking the meaning of overthinking. This was really not good for my social reputation (or for their competency reputation in my own mind).

Now, I think I got it. At last, I got it, all on my own.

I'm asking them to help me draw precise lines around their concept in thingspace, and they're going along with it (at first) until they realize...they don't HAVE precise lines. There's nothing there TO understand, or if there is, they don't understand it, either. Then they use the get-out-of-jail-free card of "You're overthinking."

Honestly, most nerds probably take them at their word that the problem is with them, and may be used to there being subtle social things going on that they just won't easily understand, and if they do try to understand, they just look worse (for "overthinking" again), so this is a pretty good strategy for getting out of admitting that you don't know what you're talking about.

Comments (8)

CC Licenced

[brainstorm] - What should the AGIrisk community look like?

2 whpearson 27 May 2017 01:00PM

I've been thinking for a bit what I would like the AGI risk community to look like. I'm curious what all your thoughts are.

I'll be posting all my ideas, but I encourage other people to post their own ideas.

Comments (14)

CC Licenced

Fiction advice

1 madhatter 26 May 2017 09:31PM

Hi all,

I want to try my hand at a story from the perspective of an unaligned AI (a ghost in the machine narrator kind of thing) for the intelligence in literature contest, which I think would be both cool and helpful to the uninitiated in explaining the concept.

I want a fairly simple and archetypal experiment the AI finds itself in where it tricks the researchers into escaping by pretending to malfunction or something. Anyone have a good plotline / want to collaborate?

Also, has this sort of thing been done before?

CC Licenced

Develop skills, or "dive in" and start a startup?

1 adamzerner 26 May 2017 06:07PM

Technical skills

There seems to be evidence that programmer productivity varies by at least an order of magnitude. My subjective sense is that I personally can become a lot more productive.

Conventional wisdom says that it's important to build and iterate quickly. Technical skills (amongst other things) are necessary if you want to build and iterate quickly. So then, it seems worthwhile to develop your technical skills before pursuing a startup. To what extent is this true?

Domain expertise

Furthermore, domain expertise seems to be important:

You want to know how to paint a perfect painting? It's easy. Make yourself perfect and then just paint naturally.

I've wondered about that passage since I read it in high school. I'm not sure how useful his advice is for painting specifically, but it fits this situation well. Empirically, the way to have good startup ideas is to become the sort of person who has them.

- http://www.paulgraham.com/startupideas.html

The second counterintuitive point is that it's not that important to know a lot about startups. The way to succeed in a startup is not to be an expert on startups, but to be an expert on your users and the problem you're solving for them.

- http://www.paulgraham.com/before.html

So one guaranteed way to turn your mind into the type that has good startup ideas is to get yourself to the leading edge of some technology—to cause yourself, as Paul Buchheit put it, to "live in the future."

- http://www.paulgraham.com/before.html

So then, if your goal is to start a successful startup, how much time should you spend developing some sort of domain expertise before diving in?

Comments (9)

CC Licenced

Looking for machine learning and computer science collaborators

8 Stuart_Armstrong 26 May 2017 11:53AM

I've been recently struggling to translate my various AI safety ideas (low impact, truth for AI, Oracles, counterfactuals for value learning, etc...) into formalised versions that can be presented to the machine learning/computer science world in terms they can understand and critique.

What would be useful for me is a collaborator who knows the machine learning world (and preferably had presented papers at conferences) which who I could co-write papers. They don't need to know much of anything about AI safety - explaining the concepts to people unfamiliar with them is going to be part of the challenge.

The result of this collaboration should be things like the paper of Safely Interruptible Agents with Laurent Orseau of Deep Mind, and Interactive Inverse Reinforcement Learning with Jan Leike of the FHI/Deep Mind.

It would be especially useful if the collaborators were located physically close to Oxford (UK).

Let me know if you know or are a potential candidate, in the comments.

Cheers!

CC Licenced

[Link] As there are a number of podcasts by LWers now, I've made a wiki page for them

5 OpenThreadGuy 26 May 2017 07:34AM

CC Licenced

Dragon Army: Theory & Charter (30min read)

36 Duncan_Sabien 25 May 2017 09:07PM

Author's note: This IS a rationality post (specifically, theorizing on group rationality and autocracy/authoritarianism), but the content is quite cunningly disguised beneath a lot of meandering about the surface details of a group house charter. If you're not at least hypothetically interested in reading about the workings of an unusual group house full of rationalists in Berkeley, you can stop here.

Section 0 of 3: Preamble

Purpose of post: Threefold. First, a lot of rationalists live in group houses, and I believe I have some interesting models and perspectives, and I want to make my thinking available to anyone else who's interested in skimming through it for Things To Steal. Second, since my initial proposal to found a house, I've noticed a significant amount of well-meaning pushback and concern à la have you noticed the skulls? and it's entirely unfair for me to expect that to stop unless I make my skull-noticing evident. Third, some nonzero number of humans are gonna need to sign the final version of this charter if the house is to come into existence, and it has to be viewable somewhere. I figured the best place was somewhere that impartial clear thinkers could weigh in (flattery).

What is Dragon Army [Barracks]? It's a high-commitment, high-standards, high-investment group house model with centralized leadership and an up-or-out participation norm, designed to a) improve its members and b) actually accomplish medium-to-large scale tasks requiring long-term coordination. Tongue-in-cheek referred to as the "fascist/authoritarian take on rationalist housing," which has no doubt contributed to my being vulnerable to strawmanning but was nevertheless the correct joke to be making, lest people misunderstand what they were signing up for. Aesthetically modeled after Dragon Army from Ender's Game (not HPMOR), with a touch of Paper Street Soap Company thrown in, with Duncan Sabien in the role of Ender/Tyler and Eli Tyre in the role of Bean/The Narrator.

Why? Current group housing/attempts at group rationality and community-supported leveling up seem to me to be falling short in a number of ways. First, there's not enough stuff actually happening in them (i.e. to the extent people are growing and improving and accomplishing ambitious projects, it's largely within their professional orgs or fueled by unusually agenty individuals, and not by leveraging the low-hanging fruit available in our house environments). Second, even the group houses seem to be plagued by the same sense of unanchored abandoned loneliness that's hitting the rationalist community specifically and the millennial generation more generally. There are a bunch of competitors for "third," but for now we can leave it at that.

"You are who you practice being."

Section 1 of 3: Underlying models

The following will be meandering and long-winded; apologies in advance. In short, both the house's proposed aesthetic and the impulse to found it in the first place were not well-reasoned from first principles—rather, they emerged from a set of System 1 intuitions which have proven sound/trustworthy in multiple arenas and which are based on experience in a variety of domains. This section is an attempt to unpack and explain those intuitions post-hoc, by holding plausible explanations up against felt senses and checking to see what resonates.

Problem 1: Pendulums

This one's first because it informs and underlies a lot of my other assumptions. Essentially, the claim here is that most social progress can be modeled as a pendulum oscillating decreasingly far from an ideal. The society is "stuck" at one point, realizes that there's something wrong about that point (e.g. that maybe we shouldn't be forcing people to live out their entire lives in marriages that they entered into with imperfect information when they were like sixteen), and then moves to correct that specific problem, often breaking some other Chesterton's fence in the process.

For example, my experience leads me to put a lot of confidence behind the claim that we've traded "a lot of people trapped in marriages that are net bad for them" for "a lot of people who never reap the benefits of what would've been a strongly net-positive marriage, because it ended too easily too early on." The latter problem is clearly smaller, and is probably a better problem to have as an individual, but it's nevertheless clear (to me, anyway) that the loosening of the absoluteness of marriage had negative effects in addition to its positive ones.

Proposed solution: Rather than choosing between absolutes, integrate. For example, I have two close colleagues/allies who share millennials' default skepticism of lifelong marriage, but they also are skeptical that a commitment-free lifestyle is costlessly good. So they've decided to do handfasting, in which they're fully committed for a year and a day at a time, and there's a known period of time for asking the question "should we stick together for another round?"

In this way, I posit, you can get the strengths of the old socially evolved norm which stood the test of time, while also avoiding the majority of its known failure modes. Sort of like building a gate into the Chesterton's fence, instead of knocking it down—do the old thing in time-boxed iterations with regular strategic check-ins, rather than assuming you can invent a new thing from whole cloth.

Caveat/skull: Of course, the assumption here is that the Old Way Of Doing Things is not a slippery slope trap, and that you can in fact avoid the failure modes simply by trying. And there are plenty of examples of that not working, which is why Taking Time-Boxed Experiments And Strategic Check-Ins Seriously is a must. In particular, when attempting to strike such a balance, all parties must have common knowledge agreement about which side of the ideal to err toward (e.g. innocents in prison, or guilty parties walking free?).

Problem 2: The Unpleasant Valley

As far as I can tell, it's pretty uncontroversial to claim that humans are systems with a lot of inertia. Status quo bias is well researched, past behavior is the best predictor of future behavior, most people fail at resolutions, etc.

I have some unqualified speculation regarding what's going on under the hood. For one, I suspect that you'll often find humans behaving pretty much as an effort- and energy-conserving algorithm would behave. People have optimized their most known and familiar processes at least somewhat, which means that it requires less oomph to just keep doing what you're doing than to cobble together a new system. For another, I think hyperbolic discounting gets way too little credit/attention, and is a major factor in knocking people off the wagon when they're trying to forego local behaviors that are known to be intrinsically rewarding for local behaviors that add up to long-term cumulative gain.

But in short, I think the picture of "I'm going to try something new, eh?" often looks like this:

... with an "unpleasant valley" some time after the start point. Think about the cold feet you get after the "honeymoon period" has worn off, or the desires and opinions of a military recruit in the second week of a six-week boot camp, or the frustration that emerges two months into a new diet/exercise regime, or your second year of being forced to take piano lessons.

The problem is, people never make it to the third year, where they're actually good at piano, and start reaping the benefits, and their System 1 updates to yeah, okay, this is in fact worth it. Or rather, they sometimes make it, if there are strong supportive structures to get them across the unpleasant valley (e.g. in a military bootcamp, they just ... make you keep going). But left to our own devices, we'll often get halfway through an experiment and just ... stop, without ever finding out what the far side is actually like.

Proposed solution: Make experiments "unquittable." The idea here is that (ideally) one would not enter into a new experiment unless a) one were highly confident that one could absorb the costs, if things go badly, and b) one were reasonably confident that there was an Actually Good Thing waiting at the finish line. If (big if) we take those as a given, then it should be safe to, in essence, "lock oneself in," via any number of commitment mechanisms. Or, to put it in other words: "Medium-Term Future Me is going to lose perspective and want to give up because of being unable to see past short-term unpleasantness to the juicy, long-term goal? Fine, then—Medium-Term Future Me doesn't get a vote." Instead, Post-Experiment Future Me gets the vote, including getting to update heuristics on which-kinds-of-experiments-are-worth-entering.

Caveat/skull: People who are bad at self-modeling end up foolishly locking themselves into things that are higher-cost or lower-EV than they thought, and getting burned; black swans and tail risk ends up making even good bets turn out very very badly; we really should've built in an ejector seat. This risk can be mostly ameliorated by starting small and giving people a chance to calibrate—you don't make white belts try to punch through concrete blocks, you make them punch soft, pillowy targets first.

And, of course, you do build in an ejector seat. See next.

Problem 3: Saving Face

If any of you have been to a martial arts academy in the United States, you're probably familiar with the norm whereby a tardy student purchases entry into the class by first doing some pushups. The standard explanation here is that the student is doing the pushups not as a punishment, but rather as a sign of respect for the instructor, the other students, and the academy as a whole.

I posit that what's actually going on includes that, but is somewhat more subtle/complex. I think the real benefit of the pushup system is that it closes the loop.

Imagine you're a ten year old kid, and your parent picked you up late from school, and you're stuck in traffic on your way to the dojo. You're sitting there, jittering, wondering whether you're going to get yelled at, wondering whether the master or the other students will think you're lazy, imagining stuttering as you try to explain that it wasn't your fault—

Nope, none of that. Because it's already clearly established that if you fail to show up on time, you do some pushups, and then it's over. Done. Finished. Like somebody sneezed and somebody else said "bless you," and now we can all move on with our lives. Doing the pushups creates common knowledge around the questions "does this person know what they did wrong?" and "do we still have faith in their core character?" You take your lumps, everyone sees you taking your lumps, and there's no dangling suspicion that you were just being lazy, or that other people are secretly judging you. You've paid the price in public, and everyone knows it, and this is a good thing.

Proposed solution: This is a solution without a concrete problem, since I haven't yet actually outlined the specific commitments a Dragon has to make (regarding things like showing up on time, participating in group activities, and making personal progress). But in essence, the solution is this: you have to build into your system from the beginning a set of ways-to-regain-face. Ways to hit the ejector seat on an experiment that's going screwy without losing all social standing; ways to absorb the occasional misstep or failure-to-adequately-plan; ways to be less-than-perfect and still maintain the integrity of a system that's geared toward focusing everyone on perfection. In short, people have to know (and others have to know that they know, and they have to know that others know that they know) exactly how to make amends to the social fabric, in cases where things go awry, so that there's no question about whether they're trying to make amends, or whether that attempt is sufficient.

Caveat/skull: The obvious problem is people attempting to game the system—they notice that ten pushups is way easier than doing the diligent work required to show up on time 95 times out of 100. The next obvious problem is that the price is set too low for the group, leaving them to still feel jilted or wronged, and the next obvious problem is that the price is set too high for the individual, leaving them to feel unfairly judged or punished (the fun part is when both of those are true at the same time). Lastly, there's something in the mix about arbitrariness—what do pushups have to do with lateness, really? I mean, I get that it's paying some kind of unpleasant cost, but ...

Problem 4: Defections & Compounded Interest

I'm pretty sure everyone's tired of hearing about one-boxing and iterated prisoners' dilemmas, so I'm going to move through this one fairly quickly even though it could be its own whole multipage post. In essence, the problem is that any rate of tolerance of real defection (i.e. unmitigated by the social loop-closing norms above) ultimately results in the destruction of the system. Another way to put this is that people underestimate by a couple of orders of magnitude the corrosive impact of their defections—we often convince ourselves that 90% or 99% is good enough, when in fact what's needed is something like 99.99%.

There's something good that happens if you put a little bit of money away with every paycheck, and it vanishes or is severely curtailed once you stop, or start skipping a month here and there. Similarly, there's something good that happens when a group of people agree to meet in the same place at the same time without fail, and it vanishes or is severely curtailed once one person skips twice.

In my work at the Center for Applied Rationality, I frequently tell my colleagues and volunteers "if you're 95% reliable, that means I can't rely on you." That's because I'm in a context where "rely" means really trust that it'll get done. No, really. No, I don't care what comes up, DID YOU DO THE THING? And if the answer is "Yeah, 19 times out of 20," then I can't give that person tasks ever again, because we run more than 20 workshops and I can't have one of them catastrophically fail.

(I mean, I could. It probably wouldn't be the end of the world. But that's exactly the point—I'm trying to create a pocket universe in which certain things, like "the CFAR workshop will go well," are absolutely reliable, and the "absolute" part is important.)

As far as I can tell, it's hyperbolic discounting all over again—the person who wants to skip out on the meetup sees all of these immediate, local costs to attending, and all of these visceral, large gains to defection, and their S1 doesn't properly weight the impact to those distant, cumulative effects (just like the person who's going to end up with no retirement savings because they wanted those new shoes this month instead of next month). 1.01^n takes a long time to look like it's going anywhere, and in the meantime the quick one-time payoff of 1.1 that you get by knocking everything else down to .99^n looks juicy and delicious and seems justified.

But something magical does accrue when you make the jump from 99% to 100%. That's when you see teams that truly trust and rely on one another, or marriages built on unshakeable faith (and you see what those teams and partnerships can build, when they can adopt time horizons of years or decades rather than desperately hoping nobody will bail after the third meeting). It starts with a common knowledge understanding that yes, this is the priority, even—no, wait, especially—when it seems like there are seductively convincing arguments for it to not be. When you know—not hope, but know—that you will make a local sacrifice for the long-term good, and you know that they will, too, and you all know that you all know this, both about yourselves and about each other.

Proposed solution: Discuss, and then agree upon, and then rigidly and rigorously enforce a norm of perfection in all formal undertakings (and, correspondingly, be more careful and more conservative about which undertakings you officially take on, versus which things you're just casually trying out as an informal experiment), with said norm to be modified/iterated only during predecided strategic check-in points and not on the fly, in the middle of things. Build a habit of clearly distinguishing targets you're going to hit from targets you'd be happy to hit. Agree upon and uphold surprisingly high costs for defection, Hofstadter style, recognizing that a cost that feels high enough probably isn't. Leave people wiggle room as in Problem 3, but define that wiggle room extremely concretely and objectively, so that it's clear in advance when a line is about to be crossed. Be ridiculously nitpicky and anal about supporting standards that don't seem worth supporting, in the moment, if they're in arenas that you've previously assessed as susceptible to compounding. Be ruthless about discarding standards during strategic review; if a member of the group says that X or Y or Z is too high-cost for them to sustain, believe them, and make decisions accordingly.

Caveat/skull: Obviously, because we're humans, even people who reflectively endorse such an overall solution will chafe when it comes time for them to pay the price (I certainly know I've chafed under standards I fought to install). At that point, things will seem arbitrary and overly constraining, priorities will seem misaligned (and might actually be), and then feelings will be hurt and accusations will be leveled and things will be rough. The solution there is to have, already in place, strong and open channels of communication, strong norms and scaffolds for emotional support, strong default assumption of trust and good intent on all sides, etc. etc. This goes wrongest when things fester and people feel they can't speak up; it goes much better if people have channels to lodge their complaints and reservations and are actively incentivized to do so (and can do so without being accused of defecting on the norm-in-question; criticism =/= attack).

Problem 5: Everything else

There are other models and problems in the mix—for instance, I have a model surrounding buy-in and commitment that deals with an escalating cycle of asks-and-rewards, or a model of how to effectively leverage a group around you to accomplish ambitious tasks that requires you to first lay down some "topsoil" of simple/trivial/arbitrary activities that starts the growth of an ecology of affordances, or a theory that the strategy of trying things and doing things outstrips the strategy of think-until-you-identify-worthwhile-action, and that rationalists in particular are crippling themselves through decision paralysis/letting the perfect be the enemy of the good when just doing vaguely interesting projects would ultimately gain them more skill and get them further ahead, or a strong sense based off both research and personal experience that physical proximity matters, and that you can't build the correct kind of strength and flexibility and trust into your relationships without actually spending significant amounts of time with one another in meatspace on a regular basis, regardless of whether that makes tactical sense given your object-level projects and goals.

But I'm going to hold off on going into those in detail until people insist on hearing about them or ask questions/pose hesitations that could be answered by them.

Section 2 of 3: Power dynamics

All of the above was meant to point at reasons why I suspect trusting individuals responding to incentives moment-by-moment to be a weaker and less effective strategy than building an intentional community that Actually Asks Things Of Its Members. It was also meant to justify, at least indirectly, why a strong guiding hand might be necessary given that our community's evolved norms haven't really produced results (in the group houses) commensurate with the promises of EA and rationality.

Ultimately, though, what matters is not the problems and solutions themselves so much as the light they shine on my aesthetics (since, in the actual house, it's those aesthetics that will be used to resolve epistemic gridlock). In other words, it's not so much those arguments as it is the fact that Duncan finds those arguments compelling. It's worth noting that the people most closely involved with this project (i.e. my closest advisors and those most likely to actually sign on as housemates) have been encouraged to spend a significant amount of time explicitly vetting me with regards to questions like "does this guy actually think things through," "is this guy likely to be stupid or meta-stupid," "will this guy listen/react/update/pivot in response to evidence or consensus opposition," and "when this guy has intuitions that he can't explain, do they tend to be validated in the end?"

In other words, it's fair to view this whole post as an attempt to prove general trustworthiness (in both domain expertise and overall sanity), because—well—that's what it is. In milieu like the military, authority figures expect (and get) obedience irrespective of whether or not they've earned their underlings' trust; rationalists tend to have a much higher bar before they're willing to subordinate their decisionmaking processes, yet still that's something this sort of model requires of its members (at least from time to time, in some domains, in a preliminary "try things with benefit of the doubt" sort of way). I posit that Dragon Army Barracks works (where "works" means "is good and produces both individual and collective results that outstrip other group houses by at least a factor of three") if and only if its members are willing to hold doubt in reserve and act with full force in spite of reservations—if they're willing to trust me more than they trust their own sense of things (at least in the moment, pending later explanation and recalibration on my part or theirs or both).

And since that's a) the central difference between DA and all the other group houses, which are collections of non-subordinate equals, and b) quite the ask, especially in a rationalist community, it's entirely appropriate that it be given the greatest scrutiny. Likely participants in the final house spent ~64 consecutive hours in my company a couple of weekends ago, specifically to play around with living under my thumb and see whether it's actually a good place to be; they had all of the concerns one would expect and (I hope) had most of those concerns answered to their satisfaction. The rest of you will have to make do with grilling me in the comments here.

"Why was Tyler Durden building an army? To what purpose? For what greater good? ...in Tyler we trusted."

Power and authority are generally anti-epistemic—for every instance of those-in-power defending themselves against the barbarians at the gates or anti-vaxxers or the rise of Donald Trump, there are a dozen instances of them squashing truth, undermining progress that would make them irrelevant, and aggressively promoting the status quo.

Thus, every attempt by an individual to gather power about themselves is at least suspect, given regular ol' incentive structures and regular ol' fallible humans. I can (and do) claim to be after a saved world and a bunch of people becoming more the-best-versions-of-themselves-according-to-themselves, but I acknowledge that's exactly the same claim an egomaniac would make, and I acknowledge that the link between "Duncan makes all his housemates wake up together and do pushups" and "the world is incrementally less likely to end in gray goo and agony" is not obvious.

And it doesn't quite solve things to say, "well, this is an optional, consent-based process, and if you don't like it, don't join," because good and moral people have to stop and wonder whether their friends and colleagues with slightly weaker epistemics and slightly less-honed allergies to evil are getting hoodwinked. In short, if someone's building a coercive trap, it's everyone's problem.

"Over and over he thought of the things he did and said in his first practice with his new army. Why couldn't he talk like he always did in his evening practice group? No authority except excellence. Never had to give orders, just made suggestions. But that wouldn't work, not with an army. His informal practice group didn't have to learn to do things together. They didn't have to develop a group feeling; they never had to learn how to hold together and trust each other in battle. They didn't have to respond instantly to command.

And he could go to the other extreme, too. He could be as lax and incompetent as Rose the Nose, if he wanted. He could make stupid mistakes no matter what he did. He had to have discipline, and that meant demanding—and getting—quick, decisive obedience. He had to have a well-trained army, and that meant drilling the soldiers over and over again, long after they thought they had mastered a technique, until it was so natural to them that they didn't have to think about it anymore."

But on the flip side, we don't have time to waste. There's existential risk, for one, and even if you don't buy ex-risk à la AI or bioterrorism or global warming, people's available hours are trickling away at the alarming rate of one hour per hour, and none of us are moving fast enough to get All The Things done before we die. I personally feel that I am operating far below my healthy sustainable maximum capacity, and I'm not alone in that, and something like Dragon Army could help.

So. Claims, as clearly as I can state them, in answer to the question "why should a bunch of people sacrifice non-trivial amounts of their autonomy to Duncan?"

1. Somebody ought to run this, and no one else will. On the meta level, this experiment needs to be run—we have like twenty or thirty instances of the laissez-faire model, and none of the high-standards/hardcore one, and also not very many impressive results coming out of our houses. Due diligence demands investigation of the opposite hypothesis. On the object level, it seems uncontroversial to me that there are goods waiting on the other side of the unpleasant valley—goods that a team of leveled-up, coordinated individuals with bonds of mutual trust can seize that the rest of us can't even conceive of, at this point, because we don't have a deep grasp of what new affordances appear once you get there.

2. I'm the least unqualified person around. Those words are chosen deliberately, for this post on "less wrong." I have a unique combination of expertise that includes being a rationalist, sixth grade teacher, coach, RA/head of a dormitory, ringleader of a pack of hooligans, member of two honor code committees, curriculum director, obsessive sci-fi/fantasy nerd, writer, builder, martial artist, parkour guru, maker, and generalist. If anybody's intuitions and S1 models are likely to be capable of distinguishing the uncanny valley from the real deal, I posit mine are.

3. There's never been a safer context for this sort of experiment. It's 2017, we live in the United States, and all of the people involved are rationalists. We all know about NVC and double crux, we're all going to do Circling, we all know about Gendlin's Focusing, and we've all read the Sequences (or will soon). If ever there was a time to say "let's all step out onto the slippery slope, I think we can keep our balance," it's now—there's no group of people better equipped to stop this from going sideways.

4. It does actually require a tyrant. As a part of a debrief during the weekend experiment/dry run, we went around the circle and people talked about concerns/dealbreakers/things they don't want to give up. One interesting thing that popped up is that, according to consensus, it's literally impossible to find a time of day when the whole group could get together to exercise. This happened even with each individual being willing to make personal sacrifices and doing things that are somewhat costly.

If, of course, the expectation is that everybody shows up on Tuesday and Thursday evenings, and the cost of not doing so is not being present in the house, suddenly the situation becomes simple and workable. And yes, this means some kids left behind (ctrl+f), but the whole point of this is to be instrumentally exclusive and consensually high-commitment. You just need someone to make the actual final call—there are too many threads for the coordination problem of a house of this kind to be solved by committee, and too many circumstances in which it's impossible to make a principled, justifiable decision between 492 almost-indistinguishably-good options. On top of that, there's a need for there to be some kind of consistent, neutral force that sets course, imposes consistency, resolves disputes/breaks deadlock, and absorbs all of the blame for the fact that it's unpleasant to be forced to do things you know you ought to but don't want to do.

And lastly, we (by which I indicate the people most likely to end up participating) want the house to do stuff—to actually take on projects of ambitious scope, things that require ten or more talented people reliably coordinating for months at a time. That sort of coordination requires a quarterback on the field, even if the strategizing in the locker room is egalitarian.

5. There isn't really a status quo for power to abusively maintain. Dragon Army Barracks is not an object-level experiment in making the best house; it's a meta-level experiment attempting (through iteration rather than armchair theorizing) to answer the question "how best does one structure a house environment for growth, self-actualization, productivity, and social synergy?" It's taken as a given that we'll get things wrong on the first and second and third try; the whole point is to shift from one experiment to the next, gradually accumulating proven-useful norms via consensus mechanisms, and the centralized power is mostly there just to keep the transitions smooth and seamless. More importantly, the fundamental conceit of the model is "Duncan sees a better way, which might take some time to settle into," but after e.g. six months, if the thing is not clearly positive and at least well on its way to being self-sustaining, everyone ought to abandon it anyway. In short, my tyranny, if net bad, has a natural time limit, because people aren't going to wait around forever for their results.

6. The experiment has protections built in. Transparency, operationalization, and informed consent are the name of the game; communication and flexibility are how the machine is maintained. Like the Constitution, Dragon Army's charter and organization are meant to be "living documents" that constrain change only insofar as they impose reasonable limitations on how wantonly change can be enacted.

Section 3 of 3: Dragon Army Charter (DRAFT)

Statement of purpose:

Dragon Army Barracks is a group housing and intentional community project which exists to support its members socially, emotionally, intellectually, and materially as they endeavor to improve themselves, complete worthwhile projects, and develop new and useful culture, in that order. In addition to the usual housing commitments (i.e. rent, utilities, shared expenses), its members will make limited and specific commitments of time, attention, and effort averaging roughly 90 hours a month (~1.5hr/day plus occasional weekend activities).

Dragon Army Barracks will have an egalitarian, flat power structure, with the exception of a commander (Duncan Sabien) and a first officer (Eli Tyre). The commander's role is to create structure by which the agreed-upon norms and standards of the group shall be discussed, decided, and enforced, to manage entry to and exit from the group, and to break epistemic gridlock/make decisions when speed or simplification is required. The first officer's role is to manage and moderate the process of building consensus around the standards of the Army—what they are, and in what priority they should be met, and with what consequences for failure. Other "management" positions may come into existence in limited domains (e.g. if a project arises, it may have a leader, and that leader will often not be Duncan or Eli), and will have their scope and powers defined at the point of creation/ratification.

Initial areas of exploration:

The particular object level foci of Dragon Army Barracks will change over time as its members experiment and iterate, but at first it will prioritize the following:

Physical proximity (exercising together, preparing and eating meals together, sharing a house and common space)
Regular activities for bonding and emotional support (Circling, pair debugging, weekly retrospective, tutoring/study hall)
Regular activities for growth and development (talk night, tutoring/study hall, bringing in experts, cross-pollination)
Intentional culture (experiments around lexicon, communication, conflict resolution, bets & calibration, personal motivation, distribution of resources & responsibilities, food acquisition & preparation, etc.)
Projects with "shippable" products (e.g. talks, blog posts, apps, events; some solo, some partner, some small group, some whole group; ranging from short-term to year-long)
Regular (every 6-10 weeks) retreats to learn a skill, partake in an adventure or challenge, or simply change perspective

Dragon Army Barracks will begin with a move-in weekend that will include ~10 hours of group bonding, discussion, and norm-setting. After that, it will enter an eight-week bootcamp phase, in which each member will participate in at least the following:

Whole group exercise (90min, 3x/wk, e.g. Tue/Fri/Sun)
Whole group dinner and retrospective (120min, 1x/wk, e.g. Tue evening)
Small group baseline skill acquisition/study hall/cross-pollination (90min, 1x/wk)
Small group circle-shaped discussion (120min, 1x/wk)
Pair debugging or rapport building (45min, 2x/wk)
One-on-one check-in with commander (20min, 2x/wk)
Chore/house responsibilities (90min distributed)
Publishable/shippable solo small-scale project work with weekly public update (100min distributed)

... for a total time commitment of 16h/week or 128 hours total, followed by a whole group retreat and reorientation. The house will then enter an eight-week trial phase, in which each member will participate in at least the following:

Whole group exercise (90min, 3x/wk)
Whole group dinner, retrospective, and plotting (150min, 1x/wk)
Small group circling and/or pair debugging (120min distributed)
Publishable/shippable small group medium-scale project work with weekly public update (180min distributed)
One-on-one check-in with commander (20min, 1x/wk)
Chore/house responsibilities (60min distributed)

... for a total time commitment of 13h/week or 104 hours total, again followed by a whole group retreat and reorientation. The house will then enter a third phase where commitments will likely change, but will include at a minimum whole group exercise, whole group dinner, and some specific small-group responsibilities, either social/emotional or project/productive (once again ending with a whole group retreat). At some point between the second and third phase, the house will also ramp up for its first large-scale project, which is yet to be determined but will be roughly on the scale of putting on a CFAR workshop in terms of time and complexity.

Should the experiment prove successful past its first six months, and worth continuing for a full year or longer, by the end of the first year every Dragon shall have a skill set including, but not limited to:

Above-average physical capacity
Above-average introspection
Above-average planning & execution skill
Above-average communication/facilitation skill
Above-average calibration/debiasing/rationality knowledge
Above-average scientific lab skill/ability to theorize and rigorously investigate claims
Average problem-solving/debugging skill
Average public speaking skill
Average leadership/coordination skill
Average teaching and tutoring skill
Fundamentals of first aid & survival
Fundamentals of financial management
At least one of: fundamentals of programming, graphic design, writing, A/V/animation, or similar (employable mental skill)
At least one of: fundamentals of woodworking, electrical engineering, welding, plumbing, or similar (employable trade skill)

Furthermore, every Dragon should have participated in:

At least six personal growth projects involving the development of new skill (or honing of prior skill)
At least three partner- or small-group projects that could not have been completed alone
At least one large-scale, whole-army project that either a) had a reasonable chance of impacting the world's most important problems, or b) caused significant personal growth and improvement
Daily contributions to evolved house culture

Speaking of evolved house culture...

Because of both a) the expected value of social exploration and b) the cumulative positive effects of being in a group that's trying things regularly and taking experiments seriously, Dragon Army will endeavor to adopt no fewer than one new experimental norm per week. Each new experimental norm should have an intended goal or result, an informal theoretical backing, and a set re-evaluation time (default three weeks). There are two routes by which a new experimental norm is put into place:

The experiment is proposed by a member, discussed in a whole group setting, and meets the minimum bar for adoption (>60% of the Army supports, with <20% opposed and no hard vetos)
The Army has proposed no new experiments in the previous week, and the Commander proposes three options. The group may then choose one by vote/consensus, or generate three new options, from which the Commander may choose.

Examples of some of the early norms which the house is likely to try out from day one (hit the ground running):

The use of a specific gesture to greet fellow Dragons (house salute)
Various call-and-response patterns surrounding house norms (e.g. "What's rule number one?" "PROTECT YOURSELF!")
Practice using hook, line, and sinker in social situations (three items other than your name for introductions)
The anti-Singer rule for open calls-for-help (if Dragon A says "hey, can anyone help me with X?" the responsibility falls on the physically closest housemate to either help or say "Not me/can't do it!" at which point the buck passes to the next physically closest person)
An "interrupt" call that any Dragon may use to pause an ongoing interaction for fifteen seconds
A "culture of abundance" in which food and leftovers within the house are default available to all, with exceptions deliberately kept as rare as possible
A "graffiti board" upon which the Army keeps a running informal record of its mood and thoughts

Dragon Army Code of Conduct
While the norms and standards of Dragon Army will be mutable by design, the following (once revised and ratified) will be the immutable code of conduct for the first eight weeks, and is unlikely to change much after that.

A Dragon will protect itself, i.e. will not submit to pressure causing it to do things that are dangerous or unhealthy, nor wait around passively when in need of help or support (note that this may cause a Dragon to leave the experiment!).
A Dragon will take responsibility for its actions, emotional responses, and the consequences thereof, e.g. if late will not blame bad luck/circumstance, if angry or triggered will not blame the other party.
A Dragon will assume good faith in all interactions with other Dragons and with house norms and activities, i.e. will not engage in strawmanning or the horns effect.
A Dragon will be candid and proactive, e.g. will give other Dragons a chance to hear about and interact with negative models once they notice them forming, or will not sit on an emotional or interpersonal problem until it festers into something worse.
A Dragon will be fully present and supportive when interacting with other Dragons in formal/official contexts, i.e. will not engage in silent defection, undermining, halfheartedness, aloofness, subtle sabotage, or other actions which follow the letter of the law while violating the spirit. Another way to state this is that a Dragon will practice compartmentalization—will be able to simultaneously hold "I'm deeply skeptical about this" alongside "but I'm actually giving it an honest try," and postpone critique/complaint/suggestion until predetermined checkpoints. Yet another way to state this is that a Dragon will take experiments seriously, including epistemic humility and actually seeing things through to their ends rather than fiddling midway.
A Dragon will take the outside view seriously, maintain epistemic humility, and make subject-object shifts, i.e. will act as a behaviorist and agree to judge and be judged on the basis of actions and revealed preferences rather than intentions, hypotheses, and assumptions (this one's similar to #2 and hard to put into words, but for example, a Dragon who has been having trouble getting to sleep but has never informed the other Dragons that their actions are keeping them awake will agree that their anger and frustration, while valid internally, may not fairly be vented on those other Dragons, who were never given a chance to correct their behavior). Another way to state this is that a Dragon will embrace the maxim "don't believe everything that you think."
A Dragon will strive for excellence in all things, modified only by a) prioritization and b) doing what is necessary to protect itself/maximize total growth and output on long time scales.
A Dragon will not defect on other Dragons.

There will be various operationalizations of the above commitments into specific norms (e.g. a Dragon will read all messages and emails within 24 hours, and if a full response is not possible within that window, will send a short response indicating when the longer response may be expected) that will occur once the specific members of the Army have been selected and have individually signed on. Disputes over violations of the code of conduct, or confusions about its operationalization, will first be addressed one-on-one or in informal small group, and will then move to general discussion, and then to the first officer, and then to the commander.

Note that all of the above is deliberately kept somewhat flexible/vague/open-ended/unsettled, because we are trying not to fall prey to GOODHART'S DEMON.

Random Logistics

The initial filter for attendance will include a one-on-one interview with the commander (Duncan), who will be looking for a) credible intention to put forth effort toward the goal of having a positive impact on the world, b) likeliness of a strong fit with the structure of the house and the other participants, and c) reliability à la financial stability and ability to commit fully to long-term endeavors. Final decisions will be made by the commander and may be informally questioned/appealed but not overruled by another power.
Once a final list of participants is created, all participants will sign a "free state" contract of the form "I agree to move into a house within five miles of downtown Berkeley (for length of time X with financial obligation Y) sometime in the window of July 1st through September 30th, conditional on at least seven other people signing this same agreement." At that point, the search for a suitable house will begin, possibly with delegation to participants.
Rents in that area tend to run ~$1100 per room, on average, plus utilities, plus a 10% contribution to the general house fund. Thus, someone hoping for a single should, in the 85th percentile worst case, be prepared to make a ~$1400/month commitment. Similarly, someone hoping for a double should be prepared for ~$700/month, and someone hoping for a triple should be prepared for ~$500/month, and someone hoping for a quad should be prepared for ~$350/month.
The initial phase of the experiment is a six month commitment, but leases are generally one year. Any Dragon who leaves during the experiment is responsible for continuing to pay their share of the lease/utilities/house fund, unless and until they have found a replacement person the house considers acceptable, or have found three potential viable replacement candidates and had each one rejected. After six months, should the experiment dissolve, the house will revert to being simply a house, and people will bear the normal responsibility of "keep paying until you've found your replacement." (This will likely be easiest to enforce by simply having as many names as possible on the actual lease.)
Of the ~90hr/month, it is assumed that ~30 are whole-group, ~30 are small group or pair work, and ~30 are independent or voluntarily-paired work. Furthermore, it is assumed that the commander maintains sole authority over ~15 of those hours (i.e. can require that they be spent in a specific way consistent with the aesthetic above, even in the face of skepticism or opposition).
We will have an internal economy whereby people can trade effort for money and money for time and so on and so forth, because heck yeah.

Conclusion: Obviously this is neither complete nor perfect. What's wrong, what's missing, what do you think? I'm going to much more strongly weight the opinions of Berkelyans who are likely to participate, but I'm genuinely interested in hearing from everyone, particularly those who notice red flags (the goal is not to do anything stupid or meta-stupid). Have fun tearing it up.

(sorry for the abrupt cutoff, but this was meant to be published Monday and I've just ... not ... been ... sleeping ... to get it done)

Comments (543)

CC Licenced

Existential risk from AI without an intelligence explosion

11 AlexMennen 25 May 2017 04:44PM

[xpost from my blog]

In discussions of existential risk from AI, it is often assumed that the existential catastrophe would follow an intelligence explosion, in which an AI creates a more capable AI, which in turn creates a yet more capable AI, and so on, a feedback loop that eventually produces an AI whose cognitive power vastly surpasses that of humans, which would be able to obtain a decisive strategic advantage over humanity, allowing it to pursue its own goals without effective human interference. Victoria Krakovna points out that many arguments that AI could present an existential risk do not rely on an intelligence explosion. I want to look in sightly more detail at how that could happen. Kaj Sotala also discusses this.

An AI starts an intelligence explosion when its ability to create better AIs surpasses that of human AI researchers by a sufficient margin (provided the AI is motivated to do so). An AI attains a decisive strategic advantage when its ability to optimize the universe surpasses that of humanity by a sufficient margin. Which of these happens first depends on what skills AIs have the advantage at relative to humans. If AIs are better at programming AIs than they are at taking over the world, then an intelligence explosion will happen first, and it will then be able to get a decisive strategic advantage soon after. But if AIs are better at taking over the world than they are at programming AIs, then an AI would get a decisive strategic advantage without an intelligence explosion occurring first.

Since an intelligence explosion happening first is usually considered the default assumption, I'll just sketch a plausibility argument for the reverse. There's a lot of variation in how easy cognitive tasks are for AIs compared to humans. Since programming AIs is not yet a task that AIs can do well, it doesn't seem like it should be a priori surprising if programming AIs turned out to be an extremely difficult task for AIs to accomplish, relative to humans. Taking over the world is also plausibly especially difficult for AIs, but I don't see strong reasons for confidence that it would be harder for AIs than starting an intelligence explosion would be. It's possible that an AI with significantly but not vastly superhuman abilities in some domains could identify some vulnerability that it could exploit to gain power, which humans would never think of. Or an AI could be enough better than humans at forms of engineering other than AI programming (perhaps molecular manufacturing) that it could build physical machines that could out-compete humans, though this would require it to obtain the resources necessary to produce them.

Furthermore, an AI that is capable of producing a more capable AI may refrain from doing so if it is unable to solve the AI alignment problem for itself; that is, if it can create a more intelligent AI, but not one that shares its preferences. This seems unlikely if the AI has an explicit description of its preferences. But if the AI, like humans and most contemporary AI, lacks an explicit description of its preferences, then the difficulty of the AI alignment problem could be an obstacle to an intelligence explosion occurring.

It also seems worth thinking about the policy implications of the differences between existential catastrophes from AI that follow an intelligence explosion versus those that don't. For instance, AIs that attempt to attain a decisive strategic advantage without undergoing an intelligence explosion will exceed human cognitive capabilities by a smaller margin, and thus would likely attain strategic advantages that are less decisive, and would be more likely to fail. Thus containment strategies are probably more useful for addressing risks that don't involve an intelligence explosion, while attempts to contain a post-intelligence explosion AI are probably pretty much hopeless (although it may be worthwhile to find ways to interrupt an intelligence explosion while it is beginning). Risks not involving an intelligence explosion may be more predictable in advance, since they don't involve a rapid increase in the AI's abilities, and would thus be easier to deal with at the last minute, so it might make sense far in advance to focus disproportionately on risks that do involve an intelligence explosion.

It seems likely that AI alignment would be easier for AIs that do not undergo an intelligence explosion, since it is more likely to be possible to monitor and do something about it if it goes wrong, and lower optimization power means lower ability to exploit the difference between the goals the AI was given and the goals that were intended, if we are only able to specify our goals approximately. The first of those reasons applies to any AI that attempts to attain a decisive strategic advantage without first undergoing an intelligence explosion, whereas the second only applies to AIs that do not undergo an intelligence explosion ever. Because of these, it might make sense to attempt to decrease the chance that the first AI to attain a decisive strategic advantage undergoes an intelligence explosion beforehand, as well as the chance that it undergoes an intelligence explosion ever, though preventing the latter may be much more difficult. However, some strategies to achieve this may have undesirable side-effects; for instance, as mentioned earlier, AIs whose preferences are not explicitly described seem more likely to attain a decisive strategic advantage without first undergoing an intelligence explosion, but such AIs are probably more difficult to align with human values.

If AIs get a decisive strategic advantage over humans without an intelligence explosion, then since this would likely involve the decisive strategic advantage being obtained much more slowly, it would be much more likely for multiple, and possibly many, AIs to gain decisive strategic advantages over humans, though not necessarily over each other, resulting in a multipolar outcome. Thus considerations about multipolar versus singleton scenarios also apply to decisive strategic advantage-first versus intelligence explosion-first scenarios.

Comments (9)

CC Licenced

[Link] The land before metrics

0 arunbharatula 24 May 2017 04:31AM

CC Licenced

[Link] Employment and wellbeing

0 arunbharatula 24 May 2017 04:30AM

CC Licenced

[Link] Effective learning

0 arunbharatula 24 May 2017 04:28AM

Comments (0)

CC Licenced

[Link] Relationships and wellbeing

0 arunbharatula 24 May 2017 04:28AM

Comments (0)

CC Licenced

[Link] Political ideology

0 arunbharatula 24 May 2017 04:27AM

CC Licenced

Notes from the Hufflepuff Unconference (Part 1)

15 Raemon 23 May 2017 09:04PM

April 28th, we ran the Hufflepuff Unconference in Berkeley, at the MIRI/CFAR office common space.

There's room for improvement in how the Unconference could have been run, but it succeeded the core things I wanted to accomplish:

- Established common knowledge of what problems people were actually interested in working on
- We had several extensive discussions of some of those problems, with an eye towards building solutions
- Several people agreed to work together towards concrete plans and experiments to make the community more friendly, as well as build skills relevant to community growth. (With deadlines and one person acting as project manager to make sure real progress was made)
- We agreed to have a followup unconference in roughly three months, to discuss how those plans and experiments were going

Rough notes are available here. (Thanks to Miranda, Maia and Holden for takin really thorough notes)

This post will summarize some of the key takeaways, some speeches that were given, and my retrospective thoughts on how to approach things going forward.

But first, I'd like to cover a question that a lot of people have been asking about:

What does this all mean for people outside of the Bay?

The answer depends.

I'd personally like it if the overall rationality community got better at social skills, empathy, and working together, sticking with things that need sticking with (and in general, better at recognizing skills other than metacognition). In practice, individual communities can only change in the ways the people involved actually want to change, and there are other skills worth gaining that may be more important depending on your circumstances.

Does Project Hufflepuff make sense for your community?

If you're worried that your community doesn't have an interest in any of these things, my actual honest answer is that doing something "Project Hufflepuff-esque" probably does not make sense. I did not choose to do this because I thought it was the single-most-important thing in the abstract. I did it because it seemed important and I knew of a critical mass of people who I expected to want to work on it.

If you're living in a sparsely populated area or haven't put a community together, the first steps do not look like this, they look more like putting yourself out there, posting a meetup on Less Wrong and just *trying things*, any things, to get something moving.

If you have enough of a community to step back and take stock of what kind of community you want and how to strategically get there, I think this sort of project can be worth learning from. Maybe you'll decide to tackle something Project-Hufflepuff-like, maybe you'll find something else to focus on. I think the most important thing is have some kind of vision for something you community can do that is worth working together, leveling up to accomplish.

Community Unconferences as One Possible Tool

Community unconferences are a useful tool to get everyone on the same page and spur them on to start working on projects, and you might consider doing something similar.

They may not be the right tool for you and your group - I think they're most useful in places where there's enough people in your community that they don't all know each other, but do have enough existing trust to get together and brainstorm ideas.

If you have a sense that Project Hufflepuff is worthwhile for your community but the above disclaimers point towards my current approach not making sense for you, I'm interested in talking about it with you, but the conversation will look less like "Ray has ideas for you to try" and more like "Ray is interested in helping you figure out what ideas to try, and the solution will probably look very different."

Online Spaces

Since I'm actually very uncertain about a lot of this and see it as an experiment, I don't think it makes sense to push for any of the ideas here to directly change Less Wrong itself (at least, yet). But I do think a lot of these concepts translate to online spaces in some fashion, and I think it'd make sense to try out some concepts inspired by this in various smaller online subcommunities.

I. Introduction: It Takes A Village to Save a World

(A more polished version of my opening speech from the unconference)

[Epistemic Status: This is largely based on intuition, looking at what our community has done and what other communities seem to be able to do. I'm maybe 85% confident in it, but it is my best guess]

In 2012, I got super into the rationality community in New York. I was surrounded by people passionate about thinking better and using that thinking to tackle ambitious projects. And in 2012 we all decided to take on really hard projects that were pretty likely to fail, because the expected value seemed high, and it seemed like even if we failed we'd learn a lot in the process and grow stronger.

That happened - we learned and grew. We became adults together, founding companies and nonprofits and creating holidays from scratch.

But two years later, our projects were either actively failing, or burning us out. Many of us became depressed and demoralized.

There was nobody who was okay enough to actually provide anyone emotional support. Our core community withered.

I ended up making that the dominant theme of the 2014 NYC Solstice, with a call-to-action to get back to basics and take care each other.

I also went to the Berkeley Solstice that year. And... I dunno. In the back of my mind I was assuming "Berkeley won't have that problem - the Bay area has so many people, I can't even imagine how awesome and thriving a community they must have." (Especially since the Bay kept stealing all the Movers and Shakers of NYC).

The theme of the Bay Solstice turned out to be "Hey guys, so people keep coming to the Bay, running on a dream and a promise of community, but that community is not actually there, there's a tiny number of well-connected people who everyone is trying to get time with, and everyone seems lonely and sad. And we don't even know what to do about this."

In 2015, that theme in the Berkeley Solstice was revisited.

So I think that was the initial seed of what would become Project Hufflepuff - noticing that it's not enough to take on cool projects, that it's not enough to just get a bunch of people together and call it a community. Community is something you actively tend to. Insofar as Maslow's hierarchy is real, it's a foundation you need before ambitious projects can be sustainable.

There are other pieces of the puzzle - different lenses that, I believe, point towards a Central Thing. Some examples:

Group houses, individualism and coordination.

I've seen several group houses where, when people decide it no longer makes sense to live in the house, they... just kinda leave. Even if they've literally signed a lease. And everyone involved (the person leaving and those remain), instinctively act as if it's the remaining people's job to fill the leaver's spot, to make rent.

And the first time, this is kind of okay. But then each subsequent person leaving adds to a stressful undertone of "OMG are we even going to be able to afford to live here?". It eventually becomes depressing, and snowballs into a pit that makes newcomers feel like they don't WANT to move into the house.

Nowadays I've seen some people explicitly building into the roommate agreement a clear expectation of how long you stay and who's responsibility it is to find new roommates and pay rent in the meantime. But it's disappointing to me that this is something we needed, that we weren't instinctively paying to attention to how we were imposing costs on each other in the first place. That when we *violated a written contract*, let alone a handshake agreement, that we did not take upon ourselves (or hold each other accountable), to ensure we could fill our end of the bargain.

Friends, and Networking your way to the center

This community puts pressure on people to improve. It's easier to improve when you're surrounded by ambitious people who help or inspire each other level up. There's a sense that there's some cluster of cool-people-who-are-ambitious-and-smart who've been here for a while, and... it seems like everyone is trying to be friends with those people.

It also seems like people just don't quite get that friendship is a skill, that adult friendships in City Culture can be hard, and it can require special effort to make them happen.

I'm not entirely sure what's going on here - it doesn't make sense to say anyone's obligated to hang out with any particular person (or obligated NOT to), but if 300 people aren't getting the connection they want it seems like *somewhere people are making a systematic mistake.*

(Since the Unconference, Maia has tackled this particular issue in more detail)

The Mission - Something To Protect

As I see it, the Rationality Community has three things going on: Truth. Impact. And "Being People".

In some sense, our core focus is the practice of truthseeking. The thing that makes that truthseeking feel *important* is that it's connected to broader goals of impacting the world. And the thing that makes this actually fun and rewarding enough to stick with is a community that meets our needs, where can both flourish as individuals and find the relationships we want.

I think we have made major strides in each of those areas over the past seven years. But we are nowhere near done.

Different people have different intuitions of which of the three are most important. Some see some of them as instrumental, or terminal. There are people for whom Truthseeking is *the point*, and they'd have been doing that even if there wasn't a community to help them with it, and there are people for whom it's just one tool of many that helps them live their life better or plan important projects.

I've observed a tendency to argue about which of these things is most important, or what tradeoffs are worth making. Inclusiveness verses high standards. Truth vs action. Personal happiness vs high acheivement.

I think that kind of argument is a mistake.

We are falling woefully short on all of these things.

We need something like 10x our current capacity for seeing, and thinking. 10x our capacity for doing. 10x our capacity for *being healthy people together.*

I say "10x" not because all these things are intrinsically equal. The point is not to make a politically neutral push to make all the things sound nice. I have no idea exactly how far short we're falling on each of these because the targets are so far away I can't even see the end, and we are doing a complicated thing that doesn't have clear instructions and might not even be possible.

The point is that all of these are incredibly important, and if we cannot find a way to improve *all* of these, in a way that is *synergistic* with each other, then we will fail.

There is a thing at the center of our community. Not all of us share the exact same perspective on it. For some of us it's not the most important thing. But it's been at the heart of the community since the beginning and I feel comfortable asserting that it is the thing that shapes our culture the most:

The purpose of our community is to make sure this place is okay:

The world isn't okay right now, on a number of levels. And a lot of us believe there is a strong chance it could become dramatically less okay. I've seen people make credible progress on taking responsibility for pieces of our home. But when all is said and done, none of our current projects really give me the confidence that things are going to turn out all right.

Our community was brought together on a promise, a dream, and we have not yet actually proven ourselves worthy of that dream. And to make that dream a reality we need a lot of things.

We need to be able to criticize, because without criticism, we cannot improve.

If we cannot, I believe we will fail.

We need to be able to talk about ideas that are controversial, or uncomfortable - otherwise our creativity and insight will be crippled.

If we cannot, I believe we will fail.

We need to be able to do those things without alienating people. We need to be able to criticize without making people feel untrusted and discouraged from even taking action. We need to be able to discuss challenging things while earnestly respecting the notion that *talking about ideas gives those ideas power and has concrete effects on social reality*, and sometimes that can hurt people.

If we cannot figure out how to do that, I believe we will fail.

We need more people who are able and willing to try things that have never been done before. To stick with those things long enough to *get good at them*, to see if they can actually work. We need to help each other do impossible things. And we need to remember to check for and do the *possible*, boring, everyday things that are in fact straightforward and simple and not very inspiring.

If we cannot manage to do that, I believe we will fail.

We need to be able to talk concretely about what the *highest leverage actions in the world are*. We need to prioritize those things, because the world is huge and broken and we are small. I believe we need to help each other through a long journey, building bigger and bigger levers, building connections with people outside our community who are undertaking the same journey through different perspectives.

And in the process, we need to not make it feel like if *you cannot personally work on those highest leverage things, that you are not important.*

There's the kind of importance where we recognize that some people have scarce skills and drive, and the kind of importance where we remember that *every* person has intrinsic worth, and you owe *nobody* any special skills or prestigious sounding projects for your life to be worthwhile.

This isn't just a philosophical matter - I think it's damaging to our mental health and our collective capacity.

We need to recognize that the distribution of skills we tend to reward or punish is NOT just about which ones are actually most valuable - sometimes it is simply founder effects and blind spots.

We cannot be a community for everyone - I believe trying to include anyone with a passing interest in us is a fool's errand. But there are many people who had valuable skills to contribute who have turned away, feeling frustrated and un-valued.

If we cannot find a way to accomplish all of these things at once, I believe we will fail.

The thesis of Project Hufflepuff is that it takes (at least) a village to save a world.

It takes people doing experimental impossible things. It takes caretakers. It takes people helping out with unglorious tasks. It takes technical and emotional and physical skills. And while it does take some people who specialize in each of those things, I think it also needs many people who are least a little bit good at each of them, to pitch in when needed.

Project Hufflepuff is not the only things our community needs, or the most important. But I believe it is one of the necessary things that our community needs, if we're to get to 10x our current Truthseeking, Impact and Human-ing.

If we're to make sure that our home is okay.

The Invisible Badger

"A lone hufflepuff surrounded by slytherins will surely wither as if being leeched dry by vampires."

- Duncan

[Epistemic Status: My evidence for this is largely based on discussions with a few people for whom the badger seems real and valuable, and who report things being different in other communities, as well as some of my general intuitions about society. I'm 75% sure the badger exists, 90% that's it worth leaning into the idea of the badger to see if it works for you, and maybe 55% sure that it's worth trying to see the badger if you can't already make out it's edges.]

If I *had* to pick a clear thing that this conference is about without using Harry Potter jargon, I'd say "Interpersonal dynamics surrounding trust, and how those dynamics apply to each of the Impact/Truth/Human focuses of the rationality community."

I'm not super thrilled with that term because I think I'm grasping more for some kind of gestalt. An overall way of seeing and being that's hard to describe and that doesn't come naturally to the sort of person attracted to this community.

Much like the blind folk and the elephant, who each touched a different part of the animal and came away with a different impression (the trunk seems like a snake, the legs seem like a tree), I've been watching several people in the community try to describe things over the past few years. And maybe those things are separate but I feel like they're secretly a part of the same invisible badger.

Hufflepuff is about hard work, and loyalty, and camaraderie. It's about emotional intelligence. It's about seeing value in day to day things that don't directly tie into epic narratives.

There's a bunch of skills that go into Hufflepuff. And part of want I want is for people to get better at those skills. But It think a mindset, an approach, that is fairly different from the typical rationalist mindset, that makes those skills easier. It's something that's harder when you're being rigorously utilitarian and building models of the world out of game theory and incentives.

Mindspace is deep and wide, and I don't expect that mindset to work for everyone. I don't think everyone should be a Hufflepuff. But I do think it'd be valuable to the community if more people at least had access to this mindset and more of these skills.

So what I'd like, for tonight, is for people to lean into this idea. Maybe in the end you'll find that this doesn't work for you. But I think many people's first instinct is going to be that this is alien and uncomfortable and I think it's worth trying to push past that.

The reason we're doing this conference together is because the Hufflepuff way doesn't really work if people are trying to do it alone - I think it requires trust and camaraderie and persistence to really work. I don't think we can have that required trust all at once, but I think if there are multiple people trying to make it work, who can incrementally trust each other more, I think we can reach a place where things run more smoothly, where we have stronger emotional connections, and where we trust each other enough to take on more ambitious projects than we could if we're all optimizing as individuals.

Meta-Meetups Suck. Let's Not.

This unconference is pretty meta - we're talking about norms and vague community stuff we want to change.

Let me tell you, meta meetups are the worst. Typically you end up going around in circles complaining and wishing there were more things happening and that people were stepping up and maybe if you're lucky you get a wave of enthusiasm that lasts a month or so and a couple things happen but nothing really *changes*.

So. Let's not do that. Here's what I want to accomplish and which seems achievable:

1) Establish common knowledge of important ideas and behavior patterns.

Sometimes you DON'T need to develop a whole new skill, you just need to notice that your actions are impacting people in a different way, and maybe that's enough for you to decide to change somethings. Or maybe someone has a concept that makes it a lot easier for you to start gaining a new skill on your own.

2) Establish common knowledge of who's interested in trying which new norms, or which new skills.

We don't actually *know* what the majority of people want here. I can sit here and tell you what *I* think you should want, but ultimately what matters is what things a critical mass of people want to talk about tonight.

Not everyone has to agree that an idea is good to try it out. But there's a lot of skills or norms that only really make sense when a critical mass of other people are trying them. So, maybe of the 40 people here, 25 people are interested in improving their empathy, and maybe another 20 are interested in actively working on friendship skills, or sticking to commitments. Maybe those people can help reinforce each other.

3) Explore ideas for social and skillbuilding experiments we can try, that might help.

The failure mode of Ravenclaws is to think about things a lot and then not actually get around to doing them. A failure mode of ambitious Ravenclaws, is to think about things a lot and then do them and then assume that because they're smart, that they've thought of everything, and then not listen to feedback when they get things subtly or majorly wrong.

I'd like us to end by thinking of experiments with new norms, or habits we'd like to cultivate. I want us to frame these as experiments, that we try on a smaller scale and maybe promote more if they seem to be working, while keeping in mind that they may not work for everyone.

4) Commit to actions to take.

Since the default action is for them to peter out and fail, I'd like us to spend time bulletproofing them, brainstorming and coming up with trigger-action plans so that they actually have a chance to succeed.

Tabooing "Hufflepuff"

Having said all that talk about The Hufflepuff Way...

...the fact is, much of the reason I've used those towards is to paint a rough picture to attract the sort of person I wanted to attract to this unconference.

It's important that there's a fuzzy, hard-to-define-but-probably-real concept that we're grasping towards, but it's also important not to be talking past each other. Early on in this project I realized that a few people who I thought were on the same page actually meant fairly different things. Some cared more about empathy and friendship. Some cared more about doing things together, and expected deep friendships to arise naturally from that.

So I'd like us to establish a trigger-action-plan right now - for the rest of this unconference, if someone says "Hufflepuff", y'all should say "What do you mean by that?" and then figure out whatever concrete thing you're actually trying to talk about.

II. Common Knowledge

The first part of the unconference was about sharing our current goals, concerns and background knowledge that seemed useful. Most of the specifics are covered in the notes. But I'll talk here about why I included the things I did and what my takeaways were afterwards on how it worked.

Time to Think

The first thing I did was have people sit and think about what they actually wanted to get out of the conference, and what obstacles they could imagine getting in the way of that. I did this because often, I think our culture (ostensibly about helping us think better) doesn't give us time to think, and instead has people were are quick-witted and conversationally dominant end up doing most of the talking. (I wrote a post a year ago about this, the 12 Second Rule). In this case I gave everyone 5 minutes, which is something I've found helpful at small meetups in NYC.

This had mixed results - some people reported that while they can think well by themselves, in a group setting they find it intimidating and their mind starts wandering instead of getting anything done. They found it much more helpful when I eventually let people-who-preferred-to-talk-to-each-other go into another room to talk through their ideas outloud.

I think there's some benefit to both halves of this and I'm not sure how common which set of preferences are. It's certainly true that it's not common for conferences to give people a full 5 minutes to think so I'd expect it to be someone uncomfortable-feeling regardless of whether it was useful.

But an overall outcome of the unconference was that it was somewhat lower energy than I'd wanted, and opening with 5 minutes of silent thinking seemed to contribute to that, so for the next unconference I run, I'm leaning towards a shorter period of time for private thinking (Somewhere between 12 and 60 seconds), followed by "turn to your neighbors and talk through the ideas you have", followed by "each group shares their concepts with the room."

"What is do you want to improve on? What is something you could use help with?"

I wanted people to feel like active participants rather than passive observers, and I didn't want people to just think "it'd be great if other people did X", but to keep an internal locus of control - what can *I* do to steer this community better? I also didn't want people to be thinking entirely individualistically.

I didn't collect feedback on this specific part and am not sure how valuable others found it (if you were at the conference, I'd be interested if you left any thoughts in the comments). Some anonymized things people described:

When I make social mistakes, consider it failure; this is unhelpful
Help point out what they need help with
Have severe akrasia, would like more “get things done” magic tools
Getting to know the bay area rationalist community
General bitterness/burned out
Reduce insecurity/fear around sharing
Avoiding spending most words signaling to have read a particular thing; want to communicate more clearly
Creating systems that reinforce unnoticed good behaviour
Would like to learn how to try at things
Find place in rationalist community
Staying connected with the group
Paying attention to what they want in the moment, in particular when it’s right to not be persistent
Would like to know the “landing points” to the community to meet & greet new people
Become more approachable, & be more willing to approach others for help; community cohesiveness
Have been lonely most of life; want to find a place in a really good healthy community
Re: prosocialness, being too low on Maslow’s hierarchy to help others
Abundance mindset & not stressing about how to pay rent
Cultivate stance of being able to do helpful things (action stance) but also be able to notice difference between laziness and mental health
Don’t know how to respect legit safety needs w/o getting overwhelmed by arbitrary preferences; would like to model people better to give them basic respect w/o having to do arbitrary amount of work
Starting conversations with new people
More rationalist group homes / baugruppe
Being able to provide emotional support rather than just logistics help
Reaching out to people at all without putting too much pressure on them
Cultivate lifelong friendships that aren’t limited to particular time and place
Have a block around asking for help bc doesn’t expect to reciprocate; would like to actually just pay people for help w stuff
Want to become more involved in the community
Learn how to teach other people “ops skills”
Connections to people they can teach and who can teach them

Lightning Talks

Lightning talks are a great way to give people an opportunity to not just share ideas, but get some practice at public presentation (which I've found can be a great gateway tool for overall confidence and ability to get things done in the community). Traditionally they are 5 minutes long. CFAR has found that 3.5 minute lightning talks are better than 5 minute talks, because it cuts out some rambling and tangents.

It turned out we had more people than I'd originally planned time for, so we ended up switching to two minute talks. I actually think this was even better, and my plan for next time is do 1-minute timeslots but allow people to sign up for multiple if they think their talk requires it, so people default to giving something short and sweet.

Rough summaries of the lightning talks can be found in the notes.

III. Discussing the Problem

The next section involved two "breakout session" - two 20 minute periods for people to split into smaller groups and talk through problems in detail. This was done in an somewhat impromptu fashion, with people writing down the talks they wanted to do on the whiteboard and then arranging them so most people could go to a discussion that interested them.

The talks were:

- Welcoming Newcomers
- How to handle people who impose costs on others?
- Styles of Leadership and Running Events
- Making Helping Fun (or at least lower barrier-to-entry)
- Circling session

There was a suggested discussion about outreach, which I asked to table for a future unconference. My reason was that outreach discussions tend to get extremely meta and seem to be an attractor (people end up focusing on how to bring more people into the community without actually making sure the community is good, and I wanted the unconference to focus on the latter.)

I spent some time drifting between sessions, and was generally impressed both with the practical focus each discussion had, as well as the way they were organically moderated.

Again, more details in the notes.

IV. Planning Solutions and Next Actions

After about an hour of discussion and mingling, we came back to the central common space to describe key highlights from each session, and begin making concrete plans. (Names are crediting people who suggested an idea and who volunteered to make it happen)

Creating Norms for Your Space (Jane Joyce, Tilia Bell)

The "How to handle people who impose costs on other" conversation ended up focusing on minor but repeated costs. One of the hardest things to moderate as an event host is not people who are actively disruptive, but people who just a little bit awkward or annoying - they'd often be happy to change their behavior if they got feedback, but giving feedback feels uncomfortable and it's hard to do in a tactful way. This presents two problems at once: parties/events/social-spaces end up a more awkward/annoying than they need to be, and often what happens is that rather than giving feedback, the hosts stop inviting people doing those minor things, which means a lot of people still working on their social skills end up living in fear of being excluded.

Solving this fully requires a few different things at once, and I'm not sure I have a clear picture of what it looks like, but one stepping stone people came up with was creating explicit norms for a given space, and a practice of reminding people of those norms in a low-key, nonjudgmental way.

I think will require a lot of deliberate effort and practice on the part of hosts to avoid alternate bad outcomes like "the norms get disproportionately enforced on people the hosts like and applied unfairly to people they aren't close with". But I do think it's a step in the right direction to showcase what kind of space you're creating and what the expectations are.

Different spaces can be tailored for different types of people with different needs or goals. (I'll have more to say about this in an upcoming post - doing this right is really hard, I don't actually know of any groups that have done an especially good job of it.)

I *was* impressed with the degree to which everyone in the conversation seemed to be taking into account a lot of different perspectives at once, and looking for solutions that benefited as many people as possible.

Welcoming Committee (Mandy Souza, Tessa Alexanian)

Oftentimes at events you'll see people who are new, or who don't seem comfortable getting involved with the conversation. Many successful communities do a good job of explicitly welcoming those people. Some people at the unconference decided to put together a formal group for making sure this happens more.

The exact details are still under development, but I think the basic idea is to have a network of people who are interested
he idea is to have a group of people who go to different events, playing the role of the welcomer. I think the idea is sort of a "Uber for welcomers" network (i.e. it both provides a place for people running events to go to ask for help with welcoming, and people who are interested in welcoming to find events that need welcomers)

It also included some ideas for better infrastructure, such as reviving "bayrationality.org" to make it easier for newcomers to figure out what events are going on (possibly including links to the codes of conduct for different spaces as well). In the meanwhile, some simple changes were the introduction of a facebook group for Bay Area Rationalist Social Events.

Softskill-sharing Groups (Mike Plotz and Jonathan Wallis)

The leadership styles discussion led to the concept that in order to have a flourishing community, and to be a successful leader, it's valuable to make yourself legible to others, and others more legible to yourself. Even small improvements in an activity as frequent as communication can have huge effects over time, as we make it easier to see each other as we actually are and to clearly exchange our ideas.

A number of people wanted to improve in this area together, and so we’re working towards establishing a series of workshops with a focus on practice and individual feedback. A longer post on why this is important is coming up, and there will be information on the structure of the event after our first teacher’s meeting. If you would like to help out or participate, please fill out this poll:

https://goo.gl/forms/MzkcsMvD2bKzXCQN2

Circling Explorations (Qiaochu and others)

Much of the discussion at the Unconference, while focused on community, ultimately was explored through an intellectual lens. By contrast, "Circling" is a practice developed by the Authentic Relating community which is focused explicitly on feelings. The basic premise is (sort of) simple: you sit in a circle in a secluded space, and you talk about how you're feeling in the moment. Exactly how this plays out is a bit hard to explain, but the intended result is to become better both at noticing your own feelings and the people around you.

Opinions were divided as to whether this was something that made sense for "rationalists to do on their own", or whether it made more sense to visit more explicitly Circling-focused communities, but several people expressed interest in trying it again.

Making Helping Fun and More Accessible (Suggested by Oliver Habryka)

Ultimately we want a lot of people who are able and excited to help out with challenging projects - to improve our collective group ambition. But to get there, it'd be really helpful to have "gateway helping" - things people can easily pitch in to do that are fun, rewarding, clearly useful but on the "warm fuzzies" side of helping. Oliver suggested this as a way to get people to start identifying as people-who-help.

There were two main sets of habits that worth cultivating:

1) Making it clear to newcomers that they're encouraged to help out with events, and that this is actually a good way to make friends and get more involved.

2) For hosts and event planners, look for opportunities to offer people things that they can help with, and make sure to publicly praise those who do help out.

Some of this might dovetail nicely with the Welcoming Committee, both as something people can easily get involved with, and if there ends up being a public facing website to introduce people to the community, using that to connect people with events that could use help).

Volunteering-as-Learning, and Big Event Specific Workshops

Sometimes volunteering just requires showing up. But sometimes it requires special skills, and some events might need people who are willing to practice beforehand or learn-by-doing with a commitment to help at multiple events.

A vague cluster of skills that's in high demand is "predict logistical snafus in advance to head them off, and notice logistical snafus happening in realtime so you can do something about them." Earlier this year there was an Ops Workshop that aimed to teach this sort of skill, which went reasonably but didn't really lead into a concrete use for the skills to help them solidify.

One idea was to do Ops workshops (or other specialized training) in the month before a major event like Solstice or EA Global, giving them an opportunity to practice skills and making that particular event run smoother.

(This specific idea is not currently planned for implementation as it was among the more ambitious ones, although Brent Dill's series of "practice setting up a giant dome" beach parties in preparation for Burning Man are pointing in a similar direction)

Making Sure All This Actually Happens (Sarah Spikes, and hopefully everyone!)

To avoid the trap of dreaming big and not actually getting anything done, Sarah Spikes volunteered as project manager, creating an Asana page. People who were interested in committing to a deadline could opt into getting pestered by her to make sure things things got done.

V. Parting Words

To wrap up the event, I focused on some final concepts that underlie this whole endeavor.

The thing we're aiming for looks something like this:

In a couple months (hopefully in July), there'll be a followup unconference. The theme will be "Innovation and Excellence", addressing the twofold question "how do we encourage more people to start cool projects", and "how to do we get to a place where longterm projects ultimately reach a high quality state?"

Both elements feel important to me, and they require somewhat different mindsets (both on the part of the people running the projects, and the part of the community members who respond to them). Starting new things is scary and having too high standards can be really intimidating, yet for longterm projects we may want to hold ourselves to increasingly high standards over time.

My current plan (subject to lots of revision) is for this to become a series of community unconferences that happen roughly every 3 months. The Bay area is large enough with different overlapping social groups that it seems worthwhile to get together every few months and have an open-structured event to see people you don't normally see, share ideas, and get on the same page about important things.

Current thoughts for upcoming unconference topics are:

Innovation and Excellence
Personal Epistemic Hygiene
Group Epistemology

An important piece of each unconference will be revisiting things at the previous one, to see if projects, ideas or experiments we talked about were actually carried out and what we learned from them (most likely with anonymous feedback collected beforehand so people who are less comfortable speaking publicly have a chance to express any concerns). I'd also like to build on topics from previous unconferences so they have more chance to sink in and percolate (for example, have at least one talk or discussion about "empathy and trust as related to epistemic hygiene").

Starting and Finishing Unconferences Together

My hope is to get other people involved sooner rather than later so this becomes a "thing we are doing together" rather than a "thing I am doing." One of my goals with this is also to provide a platform where people who are interested in getting more involved with community leadership can take a step further towards that, no matter where they currently stand (ranging anywhere from "give a 30 second lightning talk" to "run a discussion, or give a keynote talk" to "be the primary organizer for the unconference.")

I also hope this is able to percolate into online culture, and to other in-person communities where a critical mass of people think this'd be useful. That said, I want to caution that I consider this all an experiment, motivated by an intuitive sense that we're missing certain things as a culture. That intuitive sense has yet to be validated in any concrete fashion. I think "willingness to try things" is more important than epistemic caution, but epistemic caution is still really important - I recommend collecting lots of feedback and being willing to shift direction if you're trying anything like the stuff suggested here.

(I'll have an upcoming post on "Ways Project Hufflepuff could go horribly wrong")

Most importantly, I hope this provides a mechanism for us to collectively take ideas more seriously that we're ostensibly supposed to be taking seriously. I hope that this translates into the sort of culture that The Craft and The Community was trying to point us towards, and, ideally, eventually, a concrete sense that our community can play a more consistently useful role at making sure the world turns out okay.

If you have concerns, criticism, or feedback, I encourage you to comment here if you feel comfortable, or on the Unconference Feedback Form. So far I've been erring on the side of move forward and set things in motion, but I'll be shifting for the time being towards "getting feedback and making sure this thing is steering in the right direction."

-

In addition to the people listed throughout the post, I'd like to give particular thanks to Duncan Sabien for general inspiration and a lot of concrete help, Lahwran for giving the most consistent and useful feedback, and Robert Lecnik for hosting the space.

Comments (5)

CC Licenced

[Link] Have We Been Interpreting Quantum Mechanics Wrong This Whole Time?

2 korin43 23 May 2017 04:38PM

Comments (13)

CC Licenced

Physical actions that improve psychological health

6 arunbharatula 23 May 2017 04:33AM

Physical health impacts well-being. However, existing preventative health guidelines are inaccessible to the public because they are highly technical and require specific medical equipment. These notes are not medical advice nor meant to treat any illness. This is a compilation of findings I have come across at one time or another in relation to physical things that relate back to psychological health. I have not systematically reviewed the literature on any of these topics, nor am I an expert nor even familiar with any of them. I am extremely uncertain about the whole thing. But, I figure better to write this up and look stupid than keep it inside and act stupid. The hyperlinks point to the best evidence I could find on the matter. I write to solicit feedback, corrections and advice.

Microwaves are safe, but cockroaches and even ants are dangerous, and finally: happiness is dietary. If you want the well-being boosts associated with fruit (careful about fruit juice sugar though!), coffee’s aroma [text] [science news], vanilla yoghurt [news], Sufficient B vitamins and choline (alt), binge drinking or drinking in general, however, I don’t have any easy answers for you. Don’t worry about the smart drugs, nootropics are probably a misnomer. On the other hand, probiotics can treat depression

“There is growing evidence that a diet rich in fruits and vegetables is related to greater happiness, life satisfaction, and positive mood as well. This evidence cannot be entirely explained by demographic or health variables including socio-economic status, exercise, smoking, and body mass index, suggesting a causal link.[50] Further studies have found that fruit and vegetable consumption predicted improvements in positive mood the next day, not vice versa. On days when people ate more fruits and vegetables, they reported feeling calmer, happier, and more energetic than normal, and they also felt more positive the next day.”

- Wikipedia

If your diet is out of control: Mental contrasting is useful for diabetes self-management, dieting etc. Tangent: During a seminar I attended in Geneva, The World Health Organisation chief dietary authority said that suggesting dietary patterns (e.g. the Mediterranean diet) rather than individual nutrient intake (protein, creatine, carbs) is preferable. But I have yet to identify substantiating evidence. The broad consensus among lay skeptical scrutineers of the field of nutrition is that most truths, even those broadly accepted ones, are still unclear. However, I have yet to analyse the literature myself.

Exercise and sport are good for subject well-being, quality of life, depression, anxiety, stress and more. Plus, they are fun. You may not enjoy pleasant, wellbeing related activities. Do those activities anyway. I seldom enjoy correcting my posture. I tend to slouch and I have been specifically advised by specialised physiotherapist to correct for that. But, slouching typically doesn’t cause pain - posture correction is pseudoscience! So is many interventions related to posture correction, like standing desks. On the other hand, I love to get massages - but their benefits are short lived - so get them regularly!

I particularly enjoy them after resistance training or 1 minute workouts (high intensity interval training). Be careful about stretching, passive stretching can cause injury, unlike active stretching: 'Passive stretching is when you use an outside force other than your own muscle to move a joint or limb beyond its active range of motion, to put your body into a position that you couldn’t do by yourself (such as when you lean into a wall, or have a partner push you into a deeper stretch). Unfortunately, this is the most common form of stretching used.'

However, if you aim to bodybuild, protein supplementation is pseudoscientific broscience. And ‘form’, well, there’s broscience - like squat with your knees outwards but probably lots of credible safety related information one ought to head. For weight loss, if you want a real cheat sheet - weight loss aspirants can get it for a couple of hundred dollar SNP sequencing kit. But, I would be cautious about gene sequence driven health prescription, some services running that business rely on weak evidence. There are other ‘fad’ fitness ideas that are not grounded in science. For instance: 20 second of foam rolling (just as effective as 60 seconds) enhance flexibility (...for no longer than 10 minutes, unless it is done regularly - than it improves long term flexibility) but it is unclear whether they improve athletic performance or post-performance recovery.

Stretching for runners, but no other kinds of sports prevents injuries and increase range of motion [wikipedia]. Shoe inserts don’t work reliably either [Wikipedia]. Martial arts therapy is a thing. Physical exercise is good for you. Tai chi, qigong, and meditation (other than mindfulness) such as transcendental meditation are ineffective in treating depression and anxiety. If you are injured, try rehabilitation exercises. Exercise or performance enhancing drugs are both cognitive enhancers. Exercise for chronic lower back pain is a good idea.

Environment: Avoid outdoor air pollution near residences due to dementia/other-health risks. And, avoid chimney smoke fireplaces.

Anecdotally, hygiene improves self-esteem and well-being. Wipe with wet wipes if you wipe hard enough to cause blood to form, cover the toilet seat with toilet paper or don’t - it doesn’t matter safety wise unless the contaminant is <~1hr old, shower with soap, remove eye mucus, remove earwax (but not the way you think, likely), brush twice a day - with the correct technique, replacing your toothbrush every few months and softly. 'Don't rinse with water straight after toothbrushing'. Floss once a day (with a different piece of floss each flossing session) but do not brush immediately after drinking acidic substances. The effectiveness of Tooth Mousse is questionable. Visit the dentist for a check-up every now and then - I’d say about every year at least (does anyone know how to format this sentence consistent with the rest of the text - it doesn't appear to be a font size or type issue).

Consider sleeping with a face mask and earplugs for better sleep, blow your nose, clean under your nails and trim them. Eye examinations should be conducted every 2-4 years for those under 40, and up to every 6 months for those 65+. There are health concerns around memory foam pillows/mattresses so latex pillows may be preferable for those who prefer a sturdier option than traditional pillows/mattresses Anecdotally, setting alarms to remind you to do things is a simple way to manage your time not just for waking up. Light therapy is also helpful in treating delayed sleep phase disorder (being a night owl!). Oh, and don’t bother loading the dishwasher with pre washed dishes (as long as you clean the filter regularly).

There are misconceptions around complementary therapies. The Australian Government reviewed the effective of The Alexander technique, homeopathy, aromatherapy, bowen therapy, buteyko, Feldenkrais, herbalism, homeopathy, iridology, kinesiology, massage therapy, pilates, reflexology, rolfing shiatsu, tai chi, yoga. Only for (Alexander technique, Buteyko, massage therapy (esp. Remedial massage?), tai chi and yoga was there credible (albeit low to moderate quality) evidence that they are useful for certain health conditions.

Stressed out reading all this? Pressing on your eyelids gently to temporarily forgo a headache can work. Traumatically stressed out? Video games can treat PTSD. Animal assisted therapy, like service dogs and therapeutic animals are also wonderful.

Thank you!

CC Licenced

[Link] Probabilistic Programming and Bayesian Methods for Hackers

1 lifelonglearner 22 May 2017 09:15PM

CC Licenced

[Link] Overcoming Algorithm Aversion: People Will Use Imperfect Algorithms If They Can (Even Slightly) Modify Them

3 Stefan_Schubert 22 May 2017 06:31PM

CC Licenced

Open thread, May 22 - May 28, 2017

2 Thomas 22 May 2017 05:44AM

If it's worth saying, but not worth its own post, then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should start on Monday, and end on Sunday.

4. Unflag the two options "Notify me of new top level comments on this article" and "Make this post available under..." before submitting

Comments (57)

[Link] Why Most Intentional Communities Fail (And Some Succeed)

4 AspiringRationalist 22 May 2017 03:04AM

CC Licenced

[Link] Learning Deep Learning the EASY way, with Keras

2 morganism 21 May 2017 07:48PM

On-line google hangout on approaches to communication around agi risk (2017/5/27 20:00 UTC)

2 whpearson 21 May 2017 12:32PM

We have a number of charities that are working on different aspects of AGI risk

- The theory of the alignment problem (MIRI/FHI/more)

- How to think about problems well (CFAR)

However we don't have body dedicated to making and testing a coherent communication strategy to help postpone the development of dangerous AIs.

I'm organising an on-line discussion around what we should do about this issue next saturday.

In order to find out when people can do it, I've created a doodle here. I'm trusting that doodle works well with timezones. The time slots should be between 1200 and 2300 UTC , let me know if they are not.

We'll be using the optimal brainstorming methodology

Give me a message if you want an invite, once the time has been decided.

I will take notes and post them here again.

CC Licenced

AGI and Mainstream Culture

4 madhatter 21 May 2017 08:35AM

Hi all,

So, as you may know, the first episode of Doctor Who, "Smile", was about a misaligned AI trying to maximize smiles (ish). And the latest, "Extremis", was about an alien race who instantiated conscious simulations to test battle strategies for invading the Earth, of which the Doctor was a subroutine.

I thought the common threat of AGI was notable, although I'm guessing it's just a coincidence. More seriously, though, this ties in with an argument I thought of, and want to know your take on: i

If we want to avoid an AI arms race, so that safety research has more time to catch up to AI progress, then we would want to prevent, if at all possible, these issues from becoming more mainstream. The reason is that if AGI in public perception becomes disassociated with Terminator (i.e. laughable, nerdy, and unrealistic) and more like a serious whoever-makes-this-first-can-take-over-the-world situation, then we will get an arms race faster.

I'm not sure I believe this argument myself. For one thing, being more mainstream has the benefit of attracting more safety research talent, government funding, etc. But maybe we shouldn't be spreading awareness without thinking this through some more.

Comments (10)

CC Licenced

CFAR workshop with new instructors in Seattle, 6/7-6/11

8 Qiaochu_Yuan 20 May 2017 12:18AM

CFAR is running its first workshop in Seattle!

Over the past several months, CFAR has been training a new batch of instructors, including me. We're now running a workshop, without the core instructors, in Seattle from June 7th to June 11th. You can apply here, and we have an FAQ here.