August 23, 2008

Good stuff in, bad stuff out

A fun ad from IBM that makes the point... (Thanks to Mark McCabe)

Posted by jmm at 12:07 AM | Comments (0) | TrackBack | Permalink »

July 25, 2008

You think you bought the music?

(This is not really an incentive design entry, just information economics more broadly. But too interesting to pass up.)

Yahoo! Music store announced yesterday it would be closing this fall. All that music you bought (well, not many people actually bought from Yahoo! Music, but still)? They are taking down the DRM servers in September, and your computer will not be able to "phone home" to get the key. The only solution: burn to CD (which of course, made DRM pretty ineffective in the first place). Apparently the same problem occurred when Microsoft and Sony announced the shuttering of their online music stores.

Conventional notions of "owning" property generally involve control over the use of that property in perpetuity (including transfer of ownership). When there are significant use restrictions and rights retained by the provider, it's licensing, not buying. This has been drummed into us over the years with software licenses (you can't take a copy of Windows off your old machine and install it on your new machine, for example). With music, I think the general sense is that we are buying it, not licensing it, however. Be that as it may, DRM imposes licensing-like restrictions, and apparently one of them is "you may not be able to listen to this music if we decide to shut down our service in the future."

Note to self: Finish burning backup CD copies of all of my iTunes music!

Posted by jmm at 07:51 AM | Comments (0) | TrackBack | Permalink »

April 12, 2008

Pollution as revenge

One of my students alerted me to a recent dramatic episode. Author and psychologist Cooper Lawrence appeared on a Fox News segment and made some apparently false statements about the Xbox game "Mass Effect", which she admitted she had never seen or played. Irate gamers shortly thereafter started posting (to Amazon) one-star (lowest possible score) reviews of her recent book that she was plugging on Fox News. Within a day or so, there were about 400 one-star reviews, and only a handful any better.

Some of the reviewers acknowledged they had not read or even looked at the book (arguing they shouldn't have to since she reviewed a game without looking at it). Many explicitly criticized her for what she said about the game, without actually saying anything about her book.

When alerted, Amazon apparently deleted most of the reviews. Its strategy apparently was to delete reviews that mentioned the name of the game, or video games at all (the book has nothing to do with video games). With this somewhat conservative strategy, the reviews remaining (68 at the moment) are still lopsidedly negative (57 one-star, 8 two-star, 3 five-star), more than I've ever noticed for any somewhat serious book, though there's no obvious way to rule these out as legitimate reviews. (I read several and they do seem to address the content of the book, at least superficially.)

Aside from being a striking, and different example of book review pollution (past examples I've noted have been about favorable reviews written by friends and authors themselves), I think this story highlights troubling issues. The gamers have, quite possibly, intentionally damaged Lawrence's business prospects: her sales likely will be lower (I know that I pay attention to review scores when I'm choosing books to buy). Of course, she arguably damaged the sales of "Mass Effect", too. Arguably, her harm was unintentional and careless (negligent rather than malicious). But she presumably is earning money by promoting herself and her writing by appearing on TV shows: is a reasonable social response to discipline her in her for negligence? (And the reviewers who have more or less written "she speaks about things she doesn't know; don't trust her as an author" may have a reasonable point: so-called "public intellectuals" probably should be guarding their credibility in every public venue if they want people to pay them for their ideas.)

I also find it disturbing, as a consumer of book reviews, but not video games, that reviews might be revenge-polluted. Though this may discipline authors in a way that benefits gamers, is it right for them to disadvantage book readers?

I wonder how long it will be (if it hasn't already happened) before an author or publisher sues Amazon for providing a nearly-open access platform for detractors to attack a book (or CD, etc.). I don't know the law in this area well enough to judge whether Amazon is liable (after all, arguably she could sue the individual reviewers for some sort of tortious interference with her business prospects), but given the frequency of contributory negligence or similar malfeasances in other domains (such as Napster and Grokster facilitating the downloading of copyrighted materials), it seems like some lawyer will try to make the case one of these days. After all, Amazon provides the opportunity for readers to post reviews in order to advance its own business interests.

Some significant risk of contributory liability could be hugely important for the problem of screening pollution in user-contributed content. If you read some of the reviews still on Amazon's site in this example, you'll see that it would not be easy to decide which of them were "illegitimate" and delete all of those. And what kind of credibility would the review service have if publishers made a habit of deciding (behind closed doors) which too-negative reviews to delete, particularly en masse. I think Amazon has done a great job of making it clear that they permit both positive and negative reviews and don't over-select the positive ones to display, which was certainly a concern I had when they first started posting reviews. But it authors and publishers can hold it liable if they let "revenge" reviews appear, I suspect it (and similar sites) will have to shut down reviewing altogether.

(Thanks to Sarvagya Kochak.)

Posted by jmm at 01:42 PM | Comments (0) | TrackBack | Permalink »

March 29, 2008

Keeping the good stuff out at Yahoo! Answers

This is, I think, an amusing and instructive tale. I'm a bit sorry to be telling it, because I have a lot of friends at Yahoo! (especially in the Research division), and I respect the organization. The point is not to criticize Yahoo! Answers, however: keeping pollution out is a hard problem for user-contributed content information services, and that their system is imperfect is a matter for sympathy, not scorn.

While preparing for my recent presentation at Yahoo! Research, I wondered whether Yahoo! Mail was still using the the Goodmail spam-reduction system (which is based on monetary incentives). I couldn't find the answer with a quick Google search, nor by searching the Goodmail and Yahoo! corporate web sites (Goodmail claims that Yahoo! is a current client, but there was no information about whether Yahoo! is actually using the service, or what impact it is having).

So, I thought, this is a great chance to give Yahoo! Answers a try. I realize the question answerers are not generally Yahoo! employees, but I figured some knowledgeable people might notice the question. Here is my question, in full:

Is Yahoo! Mail actually using Goodmail's Certified Email? In 2005 Yahoo!, AOL and Goodmail announced that the former 2 had adopted Goodmail's "Certified Email" system to allow large senders to buy "stamps" to certify their mail (see e.g., http://tinyurl.com/2atncr). The Goodmail home page currently states that this system is available at Yahoo!. Yet I can find nothing about it searching Yahoo!Mail Help, etc. My question: I the system actually being used at Yahoo!Mail? Bonus: Any articles, reports, etc. about its success or impacts on user email experience?

A day later I received the following "Violation Notice" from Yahoo! Answers:

You have posted content to Yahoo! Answers in violation of our Community Guidelines or Terms of Service. As a result, your content has been deleted. Community Guidelines help to keep Yahoo! Answers a safe and useful community, so we appreciate your consideration of its rules.

So, what is objectionable about my question? It is not profane or a rant. It is precisely stated (though compound), and I provided background context to aid answerers (and so they knew what I already knew).

I dutifully went and read the Community Guidelines (CG) and the Terms of Service (TOS), and I could not figure out what I had violated. I had heard elsewhere that some people did not like TinyURLs because it it not clear where you are being redirected, and thus it might be used to maliciously direct traffic. But I saw nothing in the CG or TOS that prohibited URLs in general, or TinyURLs specifically.

So I contacted the link they provided to appeal the deletion. A few days later I received a reply that cut-and-pasted the information from the Yahoo! Answers help page explaining why content is deleted. This merely repeated what I had been told in the first message (since none of the other categories applied): my content was in violation of the CG or TOS. But no information was provided (second time) on how the content violated these rules.

Another address was provided to appeal the decision, so I wrote a detailed message to that address, explaining my question, and my efforts to figure out what I was violating. A few days later, I got my third email from Yahoo! Answers:

We have reviewed your appeal request. Upon review we found that your content was indeed in violation of the Yahoo! Answers Community Guidelines, Yahoo! Community Guidelines or the Yahoo! Terms of Service. As a result, your content will remain removed from Yahoo! Answers.

Well... Apparently it's clear to others that my message violates the CG or the TOS, but no one wants to tell me what the violation actually is. Three answers, all three with no specific explanation. Starting to feel like I'm a character in a Kafka novel.

At this point, I laughed and gave up (it was time for me to travel to Yahoo! to give my -- apparently dangerous and community-guideline-violating -- presentation anyway).

I have to believe that there is something about the use of a URL, a TinyURL, or the content to which I pointed that is a violation. I've looked, and found many answers that post URLs (not surprisingly) to provide people with further information. Perhaps the problem is that I was linking to a Goodmail press release on their web site, and they have a copyright notice on that page? But does Yahoo! really think providing a URL is "otherwise make available any Content that infringes any patent, trademark, trade secret, copyright" (from the TOS)? Isn't that what Yahoo's search engine does all the time?

End of story.

Moral? Yahoo! Answers is a user-contributed content platform. Like most, that means it is fundamentally an open-access publishing platform. There will be people who want to publish content that is outside the host's desired content scope. How to keep out the pollution? Yahoo! uses a well-understood, expensive method to screen: labor. People read the posted questions and make determinations about acceptability. But, as with any screen, there are Type I (false negative) and Type II (false positive) errors. Screening polluting content is hard.

(My question probably does violate something, but surely the spirit of my question does not. I had a standard, factual, reference question, ironically, to learn a fact that I wanted to use in a presentation to Yahoo! Research. A bit more clarity about what I was violating and I would have contributed desirable content to Yahoo! Answers. Instead, a "good" contributor was kept out.)

Posted by jmm at 10:19 AM | Comments (5) | TrackBack | Permalink »

Presentation at Yahoo! Research on user-contributed content

Yahoo! Research invited me to speak in their "Big Thinkers" series at the Santa Clara campus on 12 March 2008. My talk was "Incentive-centered design for user-contributed content: Getting the good stuff in, Keeping the bad stuff out."

My hosts wrote a summary of the talk (that is a bit incorrect in places and skips some of the main points, but is reasonably good), and posted a video they took of the talk. The video, unfortunately, focuses mostly on me without my visual presentation, panning only occasionally to show a handful of the 140 or so illustrations I used. The talk is, I think, much more effective with the visual component. (In particular, it reduces the impact of the amount of time I spend glancing down to check my speaker notes!)

In the talk I present a three-part story: UCC problems are unavoidably ICD problems; ICD offers a principled approach to design; and ICD works in practical settings. I described three main incentives challenges for UCC design: getting people to contribute; motivating quality and variety of contributions; and discouraging "polluters" from using the UCC platform as an opportunity to publish off-topic content (such as commercial ads, or spam). I illustrated with a number of examples in the wild, and a number of emerging research projects on which my students and I are working.

Posted by jmm at 10:02 AM | Comments (0) | Permalink »

March 03, 2008

UCC incentives the old-fashioned way

Ben Kaufman announced Kluster at TED 2008. This is a business through which businesses can solicit user-contributed content: innovative technology or product ideas, business solutions, etc. Why would anyone give a for-profit company good innovation ideas? For a cash incentive...Business post challenges with a cash bonus, and Kluster has a scheme for paying out tha bonus to people whose ideas are successful. (It also runs a prediction market on the side for wagers on which of the proposed ideas will succeed.) No volunteers here: this UCC is compensated in the traditional form of tournament prizes.

Two similar businesses, at least, are already operating: InnoCentive and Cambrian House.

Think you're smart, but don't have time or capital to turn your ideas into businesses? Go sell your ideas online!

(Based on reporting in Putting Innovation in the Hands of a Crowd - New York Times)

Posted by jmm at 01:27 AM | TrackBack | Permalink »

March 02, 2008

Looking for (well-paid, highly-trained, very busy) volunteers

The Peer to Patent project is one of my favorite examples of a user-contributed content (UCC) project recently, not because it has been very successful (yet), but because it demonstrates the surprising and important ways that UCC may go to benefit society. It's no all Wikipedia and social networking!

Peer to Patent is a project started by Prof. Beth Noveck and her Do Tank group at NYU Law School. The US Patent Office adopted it for a one-year pilot starting 15 June 2007. It is a system to post patent applications for public comment, in particular seeking suggestions about possible prior art, to assist USPTO examiners determine whether a patent should be granted. It was motivated by a widely held sense, particular in the area of software and business process patents, that the USPTO has been overwhelmed with the number of applications and the advances in technology in recent years, and that many and patents have been granted which can have the effect of stifling new innovation. During the first six months of the pilot, over 1800 people have registered to participate, and over 150 prior art references have been submitted on 24 patent applications that can be reviewed through this system.

In the February 2008 issue of the Communications of the ACM, Andy Oram published a column about the project in which he discussed the incentives challenges that may stand in the way of success. First, of course, not just any user is likely to be able to make quality contributions: to be useful, a contributor must have serious expertise in the area of the patent in order to be able to understand the application well enough to recognize possible prior art, and must know the literature well enough to identify the prior art. That's not a lot of people, and they aren't the type who have a lot of underpaid hours to volunteer. Indeed, he quotes Jon Bentley of Avaya Labs who points out that the whole essence of patenting is making money, and that the people in the best position to contribute may be those least interested in doing so.

One of the hopes of the project is that it is the monetary incentive itself -- not provided by Peer to Patent, but indirectly -- that will induce people to contribute: competitors. That is, if some company is using technology on which a patent is proposed, or is developing something similar, it will have a financial interest in seeing that the patent is not granted. Thus, they might be the ones to put the time in to review the application and propose the prior art. Although they are interested parties, as Oram says "prior art is prior art no matter who finds it".

Interesting problem, and I'm looking forward to seeing whether or not Peer to Patent can succeed (and I hope it does, because I tend to think that too many software and business patent applications are approved).

Posted by jmm at 09:28 PM | Comments (1) | TrackBack | Permalink »

February 26, 2008

Encyclopedia of Life

The Encyclopedia of Life is a rather new project to create an online and every growing encyclopedia of the species of life on earth. This is just a narrow slice of what Wikipedia nominally covers (everything!), but its ambition highlights the fact that we should expect to see more and more specialized encyclopedias growing through user-contributed content (that is, Wikipedia cannot be a successful single source of knowledge).

The task: there are currently about 1.8 million known species; the relevant scientific community thinks there are about 18 million more to be discovered. And an encyclopedia does not merely name the speciies: it compiles a wide range of useful information (some of the current developed pages have 20 or more subheadings, multiple photographs, extensive bibliographies, even references to the species in literature; see, e.g., Eastern White Pine).

And what about the usual incentives problems: why contribute? what make the effort necessary for quality contributions? how to limit pollution? By focusing on a specialized and visible community of scientists, some of these problems may be smaller than in other settings. In particular, I expect that quality will be handled by a mix of contributors not wanting to look foolish to their colleagues, and other contributors delighting in showing how much better their knowledge is as they make quality-improving edits. The rewards are similar to the standard rewards of recognition and satisfaction with documenting and adding to knowledge that have kept academia moving for the past several hundred years.

Pollution and quality also will be managed, it appears, by having volunteer experts assigned as "curators", or moderators for each page. This is reminiscent of the method that seems to work well on Slashdot, for example.

But what about inducing contributions in the first place? The project's Executive Director, Prof. James Edwards, said

“We have not given enough thought to the people who provide the information on which the Encyclopedia of Life is built. “We are looking into ways to keep that community going.”

(This quote and other material above from today's New York Times article on the EOL.)

Indeed, in a twist not often heard when talking about getting people to contribute to, say, Wikipedia, the founders are worried about the community dying off:

"The ranks of taxonomists — the scientists who describe species and revise old descriptions — have been shrinking steadily for decades. Dr. Wilson hopes the Encyclopedia of Life will foster the growth of that group."
(Carl Zimmer, NYT 26 Feb 2008.)

My student, Lian Jian, is currently working on a project to discover reasons why contributors "exit" (stop contributing to) Wikipedia. Here is a poster describing her preliminary work. As the many new and exciting user-contributed content projects mature, inducing a flow of new contributors to replace those who exit, and passing along the organizational memory and routines, will become important determinants of long-term success.

Posted by jmm at 10:08 AM | Comments (0) | TrackBack | Permalink »

January 13, 2008

All user-contributed, all the time (almost)

I've been fascinated for the past couple of years with businesses that rely on user-contributed content (UCC) for substantial inputs to production. It is sometimes jokingly referred to as the "Tom Sawyer business model": get your friends to whitewash the fence for you, without paying them (in fact, they paid Tom quite handsomely, including "a key that wouldn't unlock anything, a fragment of chalk...and a dead rat on a string"). Tom Sawyer's Whitewash

Randall Stross writes in today's New York Times about two fairly well-known businesses that have nearly perfected the art: Plenty of Fish, and Craigslist. Craigslist is a wide-open classified advertising service where employers post jobs, homeowners sell their old "Monopoly -- Star Wars Version" games and unwanted gifts, and, most piquantly, people of every shape, age, color and preference seek partners for a nearly infinite variety of polymorphously perverse, chaste and romantic interactions. Craigslist is one of the top 10 visited English language sites, has versions for 450 localities in over 50 countries, and runs with only 25 employees. All of the content is written, edited (such as it is) and maintained voluntarily by users; user volunteers also provide most of the customer service through help forums.

Plenty of Fish is more specialized and not quite as successful, but perhaps more remarkable. It is a dating service localized to 50 Canadian, US and Australian cities. Markus Frind created it and devotes only about 10 hours a week to running it...and he only in the past year hired his first employee. Yet the site has 600,000 registered users (which grows rapidly despite purging 30,000 inactives a month), and receives 50,000 new photos per day. Spam-filtering of text is done by software. Filtering of photos (to make sure they are human and clothed) is done by user volunteere: in the past year the top 120 volunteers scanned over 100,000 photos each! The users provide the customer service too, through help forums.

Great business model: have the users whitewash the fence, and you work 10 hours a week for $10 million in annual profits (Stross estimates that Frind's claim about his advertising-only profits is plausible). What are the generalizable principles. How can *I* start such a business and succeed (the road is littered with UCC-driven businesses that never turn a profit).

It is obvious that one of the most important questions is why? Why would users volunteer the time and effort to provide the content, the customer service, the photo filtering, etc.? You may think it's obvious why users want to visit Plenty of Fish: there are a lot of lonely hearts out there. And it is 100% free to users: Frind only charges advertisers. Of course, without user effort, it won't succeed: there will be no information about potential life partners, no help information, and lots of undesirable photos polluting the service. But no individual user needs to contribute anything: there is no requirement for volunteer hours (as there is at our local food coop), there is no public tracking of effort and peer pressure to pull your weight. It's a free-rider's dream.

Contributing content is easy: if you don't submit a profile you aren't going to get any dates. But what about photo scanning? Yes, you want to scan photos anyway: that's why you're there. But why not let someone else filter out the junk so you only have to filter the worthwhile photos? Is there that much of a first-mover advantage that you are willing to filter 100,000 photos per year to have a shot at being the first to contact the newest hunk? My guess is that the expected return on that investment is pretty low.

And why spend your time providing free help service to other users? Maybe Plenty of Fish is lucky to have a demographic for whom the value of time is unusually low (lonely single people with nothing else to do on Saturday night), but that just means the cost is lower to make the contribution: what is the benefit? Is it that the volunteer helpers are trying to be noticed as helpful, well-informed web geeks as a way of attracting dates?

I think the answers to these questions are transparently not obvious. If the answers were easy, we'd have a lot more people working 10 hours a week to make $10 million per year. And the answers are not likely to be something that involves only traditional economic views about incentives and motivations. Developing generalizable principles about the motivations for user-contributed content will surely need to draw on psychological explanations as well, from the psychology of personality and self, and social psychology (at least).

Posted by jmm at 11:03 AM | Permalink »

January 07, 2008

UCC search arrives...manipulation and pollution to follow soon

Jimmy Wales announced the release of the public "alpha" of his new, for-profit search service, Wikia Search. The service is built on a standard search engine, but its primary feature is that users can evaluate and comment on search results, building a user-contributed content database that Wikia hopes will improve search quality, making Wikia a viable but open (and hopefully profitable) alternative to Google.

Miguel Helft, writer for the New York Times was quick to note that such a search service might be quite vulnerable to manipulation:

Like other search engines and sites that rely on the so-called “wisdom of crowds,” the Wikia search engine is likely to be susceptible to people who try to game the system, by, for example, seeking to advance the ranking of their own site. Mr. Wales said Wikia would attempt to “block them, ban them, delete their stuff,” just as other wiki projects do.

The tension is interesting: Wikia promotes itself as a valuable alternative to Google largely because its search and ranking algorithms are open, so that users know more about why some sites are being selected or ranked more highly than others.

“I think it is unhealthy for the citizens of the world that so much of our information is controlled by such a small number of players, behind closed doors,” [Wales] said. “We really have no ability to understand and influence that process.”

But, although the search and ranking algorithms may be public, whether or not searches are being manipulated by user contributed content will not be so obvious. It is far from obvious which approach is more dependable and "open". Wikia's success apparently will depend on its ad hoc and technical methods for "blocking, banning and deleting" manipulation.

Posted by jmm at 09:23 AM | Permalink »

March 24, 2007

Getting good stuff in: Participation Inequality

An interesting phenomenon, noted by many, is that most content in user-contributed content venues (including online communities that focus more on "community" than on creating a durable information resource) is provided by a small fraction of users. Many have documented that participation in a wide variety of voluntary settings follows a power law (that the amount of contribution decreases proportional to 1 over the rank of the contributor).

Jakob Nielsen offers a nice summary including some historical references:

In most online communities, 90% of users are lurkers who never contribute, 9% of users contribute a little, and 1% of users account for almost all the action.

This sort of participation inequality has been seen in online communities, Amazon book reviews, Wikipedia edits, blogs, and peer-to-peer file sharing networks, to name a few venues (for the latter, see E. Adar and B. Huberman, "Free Riding on Gnutella", First Monday, 5 (2000), and S. Sariou, P. Gummadi, and S. Gribble, "A measurement study of peer-to-peer file sharing
systems", Multimedia Computing and Networking (January 2002).)

Inequality isn't directly an issue of "getting good stuff in", as much as about getting stuff in at all. But of course, the quality of contributions is going to depend on who is motivated to contribute, not just how many contributors there are. Thus, the problem of getting critical mass for a user-contributed content service is not just getting enough contributors, but getting the right contributors.

Posted by jmm at 01:49 PM | Permalink »

March 17, 2007

Making a business of user-contributed content

Just how important is it to manage the incentive-centered design problems for a user-contributed content web site? If you want to make a business out of it, crucial. There is not a lot of advertising revenue to be made (yet) by most online services: to succeed, you need to keep your costs very low and your page views high. ICD is critical to both.

Dan Mitchell today wrote in the New York Times about the economics of advertising-supported web sites. He reports on a study by venture capitalist Jeremy Liew, "Three ways to build an online media business to $50m in revenue", plus a followup. Liew argues that to make $50 million in revenues, you need to get to Top 10, 25 or 125 levels of US website traffic (depending on the demographic of your viewers, which affects what advertising you can deliver). The basic implication: few sites will make major revenues from advertising, and to do so is very hard.

And, so, the payoff to terrific ICD; actually the two payoffs: higher quality (more traffic) at lower cost. To get significant advertising revenues you'll need consistent high quality to attract visitors and keep them returning. That means getting good stuff in, and keeping bad stuff out. And, given the modest advertising revenues that are available (much less than $50 million is going to be possible for most sites), you'll want to keep costs down, which means getting most of your labor for free (user-contributed content) without a lot of expensive editorial staff or other interventions.

As an example of the possibility of keeping down the cost side, the Wikimedia Foundation (which owns Wikipedia) currently has less than ten full-time employees.


Posted by jmm at 08:47 AM | Permalink »

February 12, 2007

Pay as much as you want?

I stumbled on a couple of online businesses that are trying a variant on shareware pricing for content and services: "pay as much as you want".

The first is Magnatune, which sells music from selected independent artists. You can stream for free (similar to various Internet "radio" stations). If you want to download a "CD" of music from an artist, Magnatune suggests a price (typically about $8), but you can pay as little as $5 or as much as $18 (they don't seem to allow you to pay more). In a USA Today story, the CEO claims that the average price paid is $8.93, not the minimum $5 allowed.

The second is LibraryThing. If you don't know it, LibraryThing is a favorite in some SI circles: it's an online book social network / book cataloguing service. You enter books you own (with the ability to keyword search a dozen or more online catalogues, like Amazon's, and the University of Michigan Library's, to get all of the bibliographic information), tag them, share or not with others, browse through others' collections via tags, etc. LibraryThing announced on Saturday that it was intrigued by Magnatune's idea, LibraryThing charges $10 for one year, or $25 lifetime, but now "suggests" those amounts but lets you choose from a range ($19 to $55 for lifetime accounts).

Shareware is somewhat different: you get to use the shareware with or without restrictions (e.g., some functions are crippled) without paying anything, then if you want to pay (usually for an unrestricted version), the price is usually fixed. Freeware is a bit closer: lots of freeware "suggests" a donation. However, with Magnatune and LibraryThing you have to pay something to get the goods or services, but the amount is (within bounds), up to the buyer.

An interesting experiment in altruism / guilty consciences (or perhaps peer pressure or peer regard? Magnatune and LibraryThing know who you are!).

Posted by jmm at 11:15 PM | Comments (0) | Permalink »

January 31, 2007

Wikipedia is in trouble

I'm going out on a limb here: unless Wikipedia comes up with a coherent contribution policy that is consistent with the economic value of its content, it will start to deteriorate.

In a widely published Associated Press story, Brian Bergstein reports that Jimmy Wales, Wikipedia founder, Board Chair Emeritus, and currently President of for-profit Wikia, Inc., blocked the account of a small entrepreneur, Gregory Kohs, who was selling his services to (openly, with attribution) write Wikipedia articles about businesses. Wales reportedly told Kohs that his MyWikiBiz was "antithetical to Wikipedia's mission", and that even posting his stories on his personal page inside Wikipedia so independent editors could grab them and insert them in the encyclopedia was "absolutely unacceptable".

Before I get into my dire forecast, what is antithetical about someone who is paid as a professional writer to prepare content, especially if he is open about that fact? There are three "fundamental" Wikipedia editorial policies with which all contributions must comply:

  1. Neutral point of view (NPOV)
  2. Verifiability
  3. No original research

The first two are relevant here. NPOV means all content "must be written from a neutral point of view (NPOV), representing fairly and without bias all significant views." Verifiability means "any reader should be able to check that material added to Wikipedia has already been published by a reliable source." Kohs stated in his corporate materials that he is committed to compliance with these two policies: he would prepare the content for interested parties, but it would be neutral and verifiable. Of course, on any particular contribution other editors might disagree and choose to revise the content, but that is the core process of Wikipedia.

The problem is deep: arguably all contributors have a subjective (non-neutral) point of view, no matter how much they may wish, and believe otherwise. What is rather remarkable about Wikipedia is how well the group editing process has worked to enforce neutrality (and verifiability) through collective action. In any case, there is no clear reason to believe a paid professional writer is going to be systematically non-neutral any more or less than a volunteer writer.

In part, this is just a simple statement about incentives. A reasonable starting point is to accept that everyone who makes the effort to research and write material for Wikipedia is doing it for some motivating reason. Research and writing take time away from other desirable activities, so unless the writer is consistently irrational, she by revealed preference believes she is getting some benefit out of writing greater than the opportunity cost of the foregone time. It follows directly that point of view might be biased by whatever is motivating a given writer. To believe otherwise is naive. Dangerously naive, for the future of Wikipedia.

Even if the "everyone is motivated by someone" argument is too subtle for some true believers in massive social altruism, there is an obvious problem with Wikipedia's position on Gregory Kohs: surely there are many, many writers who are being paid for time and effort they devote to Wikipedia, but who are not being open about it. For example, employees of corporations, non-profits, educational institutions, etc., asked to maintain a Wikipedia entry on the corporation, who do so from an IP address not traceable to the corporation (e.g., from home). We already know from past experience that political operatives have made sub rosa contributions.

So, the problem of distinguishing between a priori neutral and a priori non-neutral contributors is deep and possibly not amenable to any reasonably effective solution. This is a fundamental problem of hidden information: the contributor knows things about her motivations and point of view that are not observable by others. Rather, others can only infer her motivations, by seeing what she writes, and at that point, the motivations are moot: if her content is not neutral or verifiable, other editors can fix it, and if she systematically violates these principles, she can be banned based on what she did, not who she purports to be.

Indeed, given the intractability of knowing the motivations and subjective viewpoints of contributors, it might seem that the sensible policy would be to encourage contributors to disclose any potential conflicts of interest, to alert editors to be vigilant for particular types of bias. This disclosure, of course, is exactly what Kohs did.

And now, for my prediction that Wikipedia is in trouble. Wikipedia has become mainstream: people in all walks of life rely on it as a valuable source of information for an enormous variety of activities. That is, the content has economic value: economic in the sense that it is a scarce resource, valuable precisely because for many purposes it is better than the next alternative (it is cheaper, or more readily available, or more reliable, or more complete, etc.). Having valuable content, of course, is the prime directive for Wikipedia, and it is, truly, a remarkable success.

However, precisely because the content has economic value to the millions of users, there are millions of agents who have an economic interest in what the content contains. Some are interested merely that content exist (for example, there are not many detailed articles about major businesses, which was the hole that Kohs was trying to plug). Others might want that content to reflect a particular point of view.

Because there is economic value to many who wish to influence the content available, they will be willing to spend resources to do the influencing. And where there are resources -- value to be obtained -- there is initiative and creativity. A policy that tries to ex ante filter out certain types of contributors based on who they are, or on very limited information about what their subjective motivations might be, is as sure to be increasingly imperfect and unsuccessful as is any spam filtering technology that tries to set up ex ante filtering rules. Sure, some of this pollution will be filtered, but there will also be false positives, and worse, those with an interest in influencing content will simply find new clever ways to get around the imperfect ex ante policies about who can contribute. And they will succeed, just as spammers in other contexts succeed, because of the intrinsic information asymmetry: the contributors know who they are and what their motivations are better than any policy rule formulated by another can ever know.

So, trying to pre-filter subjective content based on extremely limited, arbitrary information about the possible motivations of a contributor will just result in a spam-like arms race: content influencers will come up with new ways to get in and edit Wikipedia, and Wikipedia's project managers will spend ever increasing amounts of time trying to fix up the rules and filters to keep them out (but they won't succeed).

This vicious cycle has always been a possibility, and indeed, we've seen examples of pollution in Wikipedia before. The reason I think the problem is becoming quite dangerous to the future of Wikipedia its very success. By becoming such a valuable source of content, content influencers will be willing to spend ever increasing amounts to win the arms race.

Wikipedia is, unavoidably (and hooray! this is a sign of the success of its mission) an economic resource. Ignoring the unavoidable implications of that fact will doom the resource to deteriorating quality and marginalization (remember Usenet?).

Ironically, at first blush there seems to be a simple, obvious alternative right at hand: let Wikipedia be Wikipedia. The marvel of the project is that the collective editorial process maintains very high quality standards. Further, by allowing people to contribute, and then evaluating their contributions, persistent abusers can be identified and publicly humiliated (as Jimmy Wales himself was when he was caught making non-neutral edits to the Wikipedia entry about himself). Hasn't Wikipedia learned its own key lessons? Let the light shine, and better the devil you know.

(Wikipedia itself offers an enlightening summary of the battle of Kohs's efforts to contribute content. This summary serves to emphasize the impossibility of Wikipedia's fantasy of pre-screening contributors.)

Posted by jmm at 12:29 AM | Comments (4) | Permalink »

January 25, 2007

Good in or bad out?

In his New York Times Circuits Newsletter, David Pogue writes about Microsoft's recent gift of $2200 laptops to about 90 bloggers who write about technology -- laptops loaded with about-to-be-released Vista and Office 2007.

Reviewers need access to the technology they are reviewing, but as Pogue notes, MS could lend the computers.

But I'm more interested in the general point Pogue makes: we live in a culture in which most journalists are trained, and managed by editors who direct them to adhere to ethical guidelines that among other things prohibit accepting gifts from subjects of stories and reviews presented as objective. But technology is moving faster than culture, and a whole new class of influential communicators has emerged -- bloggers -- who for the most part are not trained or managed to follow a specific code of ethics.

If bloggers want durable credibility and success, the culture (theirs and the greater context in which they are embedded) will need to evolve practices and standards that establish and maintain trust. Without trust -- especially at blogs that specialize in providing information for costly decisions, like purchasing consumer electronics and software -- bloggers will lose their audiences. The speed of the development of reliable practices and reputation mechanisms may determine which parts of the blogosphere succeed, and whether much of it degenerates into a morass of spam-like paid (but disguised) product placement announcements.

Posted by jmm at 04:10 PM | Permalink »

December 24, 2006

Spamming Web 2.0

The New York Times today ran a short note highlighting CNET's story about commercial spamming of Digg.com and similar sites. There are companies being paid upwards of $15,000 to get a product placed on the front page of Digg, and most recently a top 30 Digger admitted that he entered an agreement to help elevate a new business to the front page of Digg (and solicited the other top 30 Diggers to participate).

The world was pretty darned excited when it discovered email (for most people, in the early 1990s). Spam followed in a big way within a year or two. It's clear to me that we're on the same trajectory with user-contributed content sites on the Web. There is an ever-increasing need for incentive-centered designs to help keep the bad stuff out.

Posted by jmm at 08:17 AM | Permalink »

December 17, 2006

Did someone mention user-contributed content?

Well, if Time magazine recognzies user-contributed content (Person of the Year: You) is the next big thing, maybe it is (or maybe it's on the way out).

Time recognizes there are incentives issues:

Who are these people? ... Who has that time and that energy and that passion? ... Sure, it's a mistake to romanticize all this any more than is strictly necessary. Web 2.0 harnesses the stupidity of crowds as well as its wisdom.

Posted by jmm at 02:49 PM | Permalink »

November 29, 2006

Research presentation: Web 2.0 and ICD

On 20 Nov 06 I gave an invited plenary Association Lecture at the Southern Economic Association Annual Conference in Charleston, SC. The title was "Getting the good stuff in, keeping the bad stuff out: Incentives and the Web". Here are the slides (not PowerPoint!).

In this talk geared to professional economists I explained the user-contributed content explosion that is one characteristic of so-called Web 2.0, and showed that this is happening through all phases of information production, organization, retrieval and use. I then discussed three fundamental economic issues that arise with user-contributed content: getting the good stuff in (private provision of public goods); keeping the bad stuff out (pollution); and evaluating the stuff (signaling, reputation). Familiar topics to the hordes who read this blog!

I finished with a simple elaboration to illustrate how ICD methods could be used to design mechanisms for dealing with these problems. The model is based on an event that occurred last spring on Digg.com.

Posted by jmm at 11:33 PM | Comments (0) | Permalink »

September 07, 2006

Digg changes algorithm to help keep bad stuff out

Is Digg rigged? JP thinks so. He offers an informal analysis indicating that several "top 30" Digg users cross-dig each other's posts frequently, which could mutually contribute to them staying in the top group of users.

For this post, I don't really care if there is a cartel of Digg users coordinating self-promotion. The story for ICDers is that Digg changed its system to try to reduce the possibility of this type of pollution.[1]

The announced goal is to "weigh a diversified group of Diggers more heavily than groups acting together." According to Digg co-founder Kevin Rose,

This algorithm update will look at the unique digging diversity of the individuals digging the story. Users that follow a gaming pattern will have less promotion weight. This doesn't mean that the story won't be promoted, it just means that a more diverse pool of individuals will be need [sic] to deem the story homepage-worthy.
As Rose notes, keeping the bad stuff out (the pollution problem I regularly discuss) is a well-known and ongoing challenge to a user-contributed content community: "we have learned a lot about the user base and how to defend digg from spam, artificial diggs, and digg fraud. It's a battle we will continue to fight and one that we don't take lightly" (id).


Notes

1There are interesting questions about the conditions under which it is in one's self-interest to cooperate with a cartel, and what the enforcement mechanisms are that enable this. In fact, my first published article concerned conditions causing international mineral cartels of the past century to succeed or fail: MacKie-Mason, Jeffrey K. and Robert S. Pindyck, "Cartel Theory and Cartel Experience in International Minerals Markets," in Energy: Markets and Regulation, Richard L. Gordon, Henry D. Jacoby and Martin B. Zimmerman, eds. Cambridge: MIT Press, 1987: 187-214.)

Posted by jmm at 02:17 AM | Comments (0) | Permalink »

August 05, 2006

Getting good -- not just more -- stuff in at Wikipedia

At the three day Wikimania conference this week, Wikipedia founder Jimmy Wales urged contributors to start focusing more on quality than on quantity.

Interesting incentives problems. The article count is a very visible sign of group accomplishment, and individuals can also make verifiable claims about the number of articles for which they were the initial creator. But what reward is there for improving the quality of entries? This seems like a case where the difference between intrinsic and extrinsic incentives may be important, particularly if designers want to induce contributors to shift along the quality-quantity axis in user-contributed content resources. Surely some, perhaps most of the rewards for quantity of contribution are also intrinsic, but for the designer, it might be easier to tweak the extrinsic rewards, disadvantaging quality.

Posted by jmm at 11:01 AM | Comments (0) | Permalink »

July 20, 2006

Incentives the old fashioned way: hard cash for user-contributed content

TechCrunch posted a story that AOL is coming on strong with incentives for user-contributed content:

A little known Digg-fact is that a relatively small group of users submit a large percentage of the stories that end up on the Digg home page. Netscape, which recently relaunched as a Digg-clone, wants to pay those top users to switch over to them. Jason Calacanis, who runs the Netscape property, wrote a post earlier today offering to pay top Digg users $1,000 a month or more to switch to Netscape and submit news there instead.

I don't find this surprising, and I expect we'll see more of it. We already know that many "information portals" pay people to provide ripped off or unreliable content to draw viewers for their ads. And of course, paying writers for content which earns ad bucks is the standard business model for magazines and newspapers. Little surprise that "new media" newspapers (like the new Netscape) are trying the same incentive design.

Also interesting, but again probably not too surprising, that the majority of successful contributions to digg.com come from a small group of users: 20% of stories from the top 20 contributors; a bit over half from the top 100 contributors.

Posted by jmm at 05:33 PM | Permalink »

June 17, 2006

Open access to Wikipedia less open

Growing Wikipedia Revises Its 'Anyone Can Edit' Policy - New York Times (registration required).

Wikipedia "organically" grows its content through the contributions of volunteers. Most of the work appears to be done by about 1000 core participants, but for the most part, anyone on the planet with Internet access can edit, delete or add content. Until recently...

This past winter, the media drew attention to a few glaring examples of subjective Wikipedia manipulation. One of the first publicized was a joke reference to John Seigenthaler Sr. as a participant in the assassinations of both John and Robert Kennedy. But then folks started noticing that politicians' pages were burnished, and other dubious content.

Now, in response, the informal Wikipedia management has implemented rules that put some limits on who may edit an article, and when. A small number of articles are "protected" so that no one can edit them (except the management team?) until further notice. Another group are "semi-protected" so that only users who have been registered for at least four days may edit. (Both protected and semi-protected articles are labeled in color at the top of the entry so that readers are warned about the controversy and the likelihood that some content is unreliable.)

For ICD, it is the latter category that is interesting. The "protected" articles simply are not open-access user-contributed content during the time that they are protected; they have become a traditional proprietary content vehicle. But the "semi-protected" articles are still open, with an incentive mechanism to discourage inappropriate editing. There are a couple of elements to the mechanism, and various ways in which these elements may interact with incentives:

Registration Only registered users may edit a semi-protected article. Since there is no identity authentication when an account is created, this mostly serves as a small cost imposed on users who want to edit. The pseudonymous identity can be used to build a positive reputation through repeated good contributions, but is not likely to do much to discourage bad contributions since the pseudonym can be abandoned. (I think, however, that the Wikipedia system also tracks the IP address that editors are using.)
Waiting period New pseudonyms have to wait four days before editing semi-protected articles. This cooling off period seems intended primarily to slow down "revert wars" when two opposing camps rapidly re-edit a controversial page. This kind of pollution is annoying, but probably not too dangerous since it is pretty evident to anyone who is making serious use of an article. The cooling off also may be enough of a time cost to discourage casual polluters who think it would be fun to manipulate an article on a controversial topic.


The NYT published a list of articles currently under editing restrictions.

Posted by jmm at 10:23 AM | Permalink »

April 08, 2006

Polluting user-contributed reviews

A recent First Monday article by David and Pinch (2006) documents an interesting case of book review pollution on Amazon. A user review of one book critically compared it to another. Immediately following a "user" entered another review blatantly plagiarizing a favorable review of the first book, and further user reviews did additional plagiarizing.

When the author of the first book discovered the plagiarism, he notified Amazon which at the time had a completely hands-off policy on user reviews, so it refused to intervene even for blatant plagiarism. (The policy since has changed.) Another example of the problem of keeping bad quality contributions out.

David and Pinch remind us that when an Amazon Canada programming glitch revealed reviewer identities,

a large number of authors had "gotten glowing testimonials from friends, husbands, wives, colleagues or paid professionals." A few had even 'reviewed' their own books, and, unsurprisingly, some had unfairly slurred the competition.

David and Pinch address the issue of review pollution at some length. First, the catalogue six discrete layers of reputation in the Amazon system, including user ratings of reviews by others, and a mechanism to report abuse. Then they conducted an analysis of 50,000 reviews of 10,000 books and CDs. Categories of review pollution they identified automatically (using software algorithms):

They also make an interesting point about the arms-race limitations of technical pollution screens:

The sorts of practices we have documented in this paper could have been documented by Amazon.com themselves (and for all we know may have indeed been documented). Furthermore if we can write an algorithm to detect copying then it is possible for Amazon.com to go further and use such algorithms to alert users to copying and if necessary remove material. If Amazon.com were to write such an algorithm and, say, remove copied material, this will not be the end of the story. Users will adapt to the new feature and will no doubt try and find new ways to game the system.

Posted by jmm at 02:35 PM | Comments (0) | Permalink »

Keeping bad stuff out: Making a play on social news sites?

About a month ago, some rumors were about that Google was about to acquire Sun Microsystems. The news got hot when blog stories claiming an acquisition was imminent were promoted to the front page on community/social new site Digg.com. It pretty quickly became clear that the rumors were largely unfounded. What hasn't been quickly resolved is whether or not someone tried to manipulate Digg, possibly to cash in on speculative trading in Google or Sun stock.

The basic idea is simple: get enough shill users to vote for a financially-significant rumor to promote it to the front page, thus automatically getting more widespread attention, and hope that the burst of attention causes a temporary stock price adjustment that can be exploited. (For example, in an acquisition the price of Sun would almost surely increase, and thus gullible readers might start buying it and bidding it up; the scam artist could purchase shares in advance to sell at the inflated price, or sell it short at the bubble price and collect when price returns to normal.)

Digg claims that it almost surely was not manipulated, but it seems clear that such manipulation is possible in user-contributed content news sites. Recall how Rich Wiggins found that people could get flim-flam press releases fed into Google News (here and here), and how authors using pseudonyms have promoted their own books with favorable "reviews" on Amazon.com.

It appears that in the past Digg has been manipulated (though apparently as an experiment, not to manipulate stock prices).

Posted by jmm at 02:05 AM | Comments (0) | Permalink »

April 05, 2006

Didn't keep the bad stuff out

Here's a user-contributed content service that sure didn't keep the bad stuff out (from their point of view!). Chevy set up an interactive marketing site at which you could create your own customized version of a video commercial for one of their SUVs. A few users had some fun with the freedom. (Good chance the links won't last at the Chevy site for long -- I'll try to find another source when they disappear.)
[link][link][link]

Posted by jmm at 10:22 PM | Comments (0) | Permalink »

March 15, 2006

i-Newswire is out, that's who

A couple of days after Rich Wiggins posted his blog story about the ability to place false news stories in Google News, CNN has picked up the story, and Google has now dropped i-Newswire as a source for Google News.

i-Newswire was a user-contributed content (UCC) service, and thus subject to the pollution problem I've been discussing (link and link). More precisely, i-Newswire is an un-moderated or un-edited UCC service (all press release newswires rely on user-contributed content, but most employ editors to decide whether press released are legitimate).

Google News, on the other hand, is not a UCC, and is edited: there is central control over which content feeds are included. So, in a crude way, Google can handle the pollution problem: if pollution is coming in through channel A, turn channel A off. Google News may be a case where a technological pollution prevention approach will work pretty well, obviating the need for an incentive system.

Posted by jmm at 10:25 PM | Comments (0) | Permalink »

March 14, 2006

Digg, Google News...User-contributed "news"

I'm developing an interest in the phenomenon of user-contributed content, and the two fundmental incentives problems that it faces: pollution (keeping the bad stuff out) and the private provision of public goods (inducing contributions of the good stuff). User-contributed "news" is one example to explore.

Digg.com is one currently hot user-contributed news site:

Digg is a technology news website that combines social bookmarking, blogging, RSS, and non-hierarchical editorial control. With digg, users submit stories for review, but rather than allow an editor to decide which stories go on the homepage, the users do.

Slashdot of course is the grande dame. Digg and Slashdot both rely on multiple techniques of community moderation to try to maintain the quality of content (keep out the pollution). For example, proposed stories for Digg are not promoted to the homepage until they have sufficient support from multiple users; and users can report bad entries (apparently to a team of human editors).

How effective (and socially costly) are these community moderation techniques? By now we've all heard about Wikipedia founder Jimmy Wales manipulating his own Wikipedia entry, which led to publicity about multiple members of Congress, etc., who have been doing the same thing.

And even if a site has an efficient moderation system to filter out pollution, there is still the problem of inducing people to volunteer time and effort to contribute to the public good by creating valuable content. Obviously, this can happen (see Slashdot, Wikipedia). But suppose you are designing a new user-contributed content service: how are you going to create a community of users, and how are you going to induce them to donate (high quality) content?

Apparently we can now start to count Google News as a site for user-contributed news.

Posted by jmm at 08:18 AM | Comments (0) | Permalink »

Spamming Google News: Who's in, who's out?

An old acquaintance of mine, Rich Wiggins, recently blogged about his discovery of how easy it is to insert content in Google News. He discovered this when he noticed regular press releases published in Google News that were a front for the musings of self-proclaimed "2008 Presidential contender" Daniel Imperato. Who?

Wiggins figured out how Imperato did it, and tested the method by publishing a press release (screen shot) about his thoughts while celebrating his 50th birthday in Florida. Sure enough, you can find this item by searching on "Rich Wiggins" in Google News.

This is (for now) a fun example of one of the two fundamental incentives problems for important and fast-growing phenomenon of user-contributed content:


  1. How to keep the undesirable stuff out?
  2. How to induce people to contribute desirable stuff?

The first we can call the pollution problem, the second the private provision of public goods problem. Though Wiggins example is funny, will we soon find Google News polluted beyond usefulness (the decline of the Usenet was largely due to spam pollution).

Blogs, of course, are a major example of user-contributed content. At first glance, they don't suffer as much from the first problem: readers know that blogs are personal, unvetted opinion pages, and so they don't blindly rely on what is posted as truth. (Or do they?) But then there's the problem of splogging, which isn't really a problem for blogs as much as for search engines that are being tricked into directing searchers to fake blog pages that are in fact spam advertisements (a commercial variant on the older practice of Google bombing).

There is a lengthy and informative Wikipedia article that discusses the wide variety of pollution techniques (spamming) that have been developed for many different settings (besides email and blogs, also instant messaging, cell phones, online games, wikis, etc.), with an index to a family of detailed articles on each subtype.

Posted by jmm at 07:44 AM | Comments (0) | Permalink »

March 03, 2006

Web 2.0 vulnerabilities

Wired ran an article last fall about vulnerabilities becoming apparent in various "Web 2.0" applications (whatever those are). Some are similar to spam in email: for example, splogs (fake blogs created to attract search engine interest and drive viewers to see their Viagra ads).

Many interesting social computing applications have enough openness that they are vulnerable to misuses and manipulations. A traditional approach is to develop technical means to close or limit the vulnerabilities (like filters for spam). We know that the inevitable trade-off between the benefits (even necessity) of some degree of openness for social applications and the resulting vulnerability means technical solutions are unlikely to be 100% satisfactory. That leaves open the room for incentive-based mechanisms to discourage misuse of social computing applications, like the various payment schemes proposed to fight spam. What incentive scheme might reduce splogging, for example?

For many social computing applications, financial incentive schemes may be undesirable, suggesting a growing need to develop effective non-pecuniary incentive mechanisms.

Posted by jmm at 10:56 PM | Comments (0) | Permalink »

March 01, 2006

Ad-supported web affects content creation incentives

For years I've been interested in the way that the financial model for information content affects the incentives to create, the quality and the diversity of the content.

The basic point is simple: if readers are paying for content (buying a book, subscribing to a service), then presumably the creators are trying to create content valuable to the readers. In the other leading model, advertisers pay for content: then presumably creators are trying to create content that attracts the attention of readers, but isn't necessarily of high value to them.

Does this explain the quality difference between typical broadcast TV shows and the subscriber content like "The Sopranos" and "Six Feet Under"?

I, together with a couple of students, gathered a lot of facts and notes on this topic several years back, but I haven't written much on it. (The idea shows up in passing in a couple of my scholarly articles.)

The Wall Street Journal published a nice column illustrating just this point for current web site content creation.

"If there is a topic in the news, people will be searching on it. If you can get those searchers to land on a seemingly authoritative page you've set up, you can make money from their arrival. Via ads, for instance. Then, to get your site ranked high in search engines, it's best to have 'original content' about whatever the subject of your site happens to be. The content needs to include all the keywords that people might search for. But it can't be just an outright copy of what's on some other site; you get penalized for that by search engines."
The WSJ author contracted as a freelance writer to create content for a site, and found that the assignment was primarily to cut-and-paste content from elsewhere with enough changes to fool the search engines.

I think there are some important ICD opportunities here for people thinking about creating content portals and other information services.

(Thanks to Rick Wash for pointing me to this column.)

Posted by jmm at 10:50 PM | Comments (0) | Permalink »