웹 서비스 Insight

Yesterday I interviewed digg founder Kevin Rose to discuss digg's popularity, its battle with spammers, the recent issues with GroupThink and digg's upcoming personalization features. This is the first of a two-part article presenting that interview. NB: Part 2 available now.

As you know, digg.com is a technology news site that over the past 6 months has begun to rival Slashdot in popularity among IT geeks. But whereas Slashdot content is controlled by editors, digg prides itself on being run by its community. Which articles make it to the coveted digg homepage is determined by how many "diggs" they get from the readers. digg calls this "non-hierarchical editorial control", in a none-to-subtle dig at Slashdot. 

We plan to add the ability to mark stories as false or inaccurate However in recent weeks there have been signs of possible cracks in this user-driven system, with some people wondering whether the 'wisdom of the masses' is more like a mob mentality at times. I myself have published a couple of posts critical of the digg system, firstly about their (what I at the time perceived to be) overzealous response to spam and then there was the highly controversial group tarring and feathering of an innocent O'Reilly writer. Both of those incidents made me question the system, although I continue to be impressed by digg's potential. 

I thought what better way to get to the bottom of these issues than to interview the people behind digg - and so I'm grateful that Kevin Rose agreed to talk to me. The interview will run over two days and here now is Part 1.

The Tipping Point

Richard: How and when did digg get started?

Kevin: We started working on developing the site back in October 2004. We started toying around with the idea a couple of months prior to that, but it was early October when we actually started creating what would become the beta version of digg. The site launched to the world on December 5th 2004.

Richard: How long did it take to gain momentum and a large user base? It's extremely popular now, but how long did it take to get to that tipping point?

Kevin: We never had any idea it would become this popular. It was just a side project of mine and something that we wanted to try out by giving power and control back to the masses. Typically with technology news sites a handful of editors choose which stories are relevant and which they believe the audience would like to read about. This was the first time that anyone experimented with allowing the general mass audience to decide what they believed to be the most important topic of the day. 

So initially there were a few months where everyone had to wrap their head around the idea of how the entire digg concept worked. Around February or March 2005 we started to see the site take off Because when you first come to the site you see the number of diggs, but initially you don't understand the benefit of digging something and how that's also bookmarked into your own personal profile. Every activity you do on the site is saved as an RSS feed and can be shared with friends. The site increases in value with each new user added to the critical mass already on digg. 

It was around February or March of 2005 that we really started to see the site take off. One of the things that pushed us into the mainstream was the Paris Hilton cellphone hack, in which her cellphone images and data were stolen and posted online. Someone posted the story on digg late one night and because of the popularity of the story it was quickly dugg onto the homepage. We were then indexed by Google and Yahoo that night under the keywords "paris hilton cellphone hack" and when we woke up the next morning, we saw our site was crawling along… because there were hundreds of thousands of people that were typing those keywords in - and the first site that would come up was digg. So it was a really strange incident at the time, but it showed us the power of using the masses to break news stories. 

It works by users finding relevant information and sending it to the digg community, which then takes that story and diggs it - and once it gains enough momentum and interest, it's promoted to the homepage for all to see.

Reputation Systems - does digg have one?

Richard: I'd like to talk now about reputation systems. I wrote a post recently about reddit's karma system, which is similar to Slashdot's reputation system. I wondered whether the same thing will be built into digg, or whether you think it's necessary even?

Kevin: Well there is a karma system that exists behind the scenes of digg.  It is basically used for spam and fraud detection. So it's an automated system where we can detect where there are people trying to abuse the system. It identifies those users and notifies an administrator of activities that violate our terms of service.  The administrator then takes appropriate action based on the nature of the violation. 

In the future when we release the next version of our comments system For the most part it's self-policing by the users we're going to allow people to gain a reputation by the ability to digg or bury comments that they like or dislike. But for the most part it's the self-policing by the users of the community that helps keep things in check. Users have the ability to report a story that they believe is irrelevant or doesn't pertain to tech or is a duplicate entry. It's the users of the community that are performing all the duties that would typically be performed by a standard admin on the site. So, for example, users can "bury" a story and when enough users do that it's removed from the queue, but not deleted from the site.

Richard: So is it fair to say then that rather than a reputation system for people, it's more of a system for the story itself? That is, you're not voting on whether the person commenting on the story is good or bad, or whatever, but you're voting on whether the story itself is good or bad. Is that a good way to look at it?

Kevin: That's a good way to look at it. Rating individual users really doesn’t scale well The problem with rating individual users is that it really doesn't scale well. We have 160,000 registered users and it would be impossible for everyone to go through and directly rate everyone else. What we have created, which is quickly becoming a popular way to discover new stories, is our friends system. This enables you to create a group of trusted users who read each others articles and trust each others content. You can combine articles and friends that you're tracking into something almost like a watchlist. You're into what they're digging, you're into what they're submitting and commenting on. And digg takes all that information under your own personal profile on digg and combines it all together in a single feed for you to pick through. 

The GroupThink Issue

Richard: That leads to the question of groupthink. There's been a bit of discussion recently about whether digg users actually add their own personal opinions on stories, or whether they just jump on board whatever train of thought the community has at the time. Do you think that's a concern for you, are you trying to address that issue? I'm thinking specifically of the incident with the O'Reilly writer - are you doing anything to improve the system in that respect?

Kevin: Since news breaks on digg very quickly, we face the same issues as newspapers which print a retraction for a story that was misreported. The difference with digg is that equal play can be given to both sides of a story, whereas with a newspaper, a retraction or correction is usually buried.

In the case that you mentioned with the O'Reilly writer, it was an inaccurate story that was promoted to the homepage, and then quickly thereafter, a correct story was submitted and made the digg homepage as well. 

We plan to add the ability to mark stories as false or inaccurate, so that other users can see they've been marked inaccurate. There's such a mass of stories coming in at a very fast rate — it's nice to know that as quickly as digg can break stories, they can also break follow-up stories that might contain additional information or a follow-up to the original story. 

Richard: Staying on the issue with the O'Reilly writer for a minute, he posted a follow-up post on his blog saying that he was concerned about the original digg article being on the first results page of Google when people do a search on his name. Is there any way for digg, or Google even, to make amends on that - because obviously that's a bit harsh on the poor O'Reilly writer…

Kevin: Within the next week, we're turning a new system live that will immediately alert readers if a story has been marked as inaccurate - the story will display that right off the bat, right at the top of the story itself. So the users will know as soon as they see the story… much like a Wikipedia entry that contains information that is potentially inaccurate, it's marked in the same way.

In Part 1 of my interview with digg founder Kevin Rose, we discussed digg's popularity, reputation systems and recent issues with GroupThink. In the final part of this interview, Kevin and I talk about digg's battle with spammers and the upcoming release of new personalization features, plus an API.

On Fighting Spam

Richard: Now to the question of spam. I know you've had a number of issues fighting spam recently, so I wondered if you would explain some of the technical ways that you are fighting this spam and making the system better in the future - in that respect.

Kevin: The way that we're fighting it the most is allowing the community to handle it. Right now approximately 1,200 news stories are submitted per day. When you have a site that gets as much traffic as digg is now starting to receive, you have users that will try and game the system. It's natural for them to try and create false accounts or send personal blog posts.  But, the digg community quickly sees through a lot of this - and users are marking these stories, and removing and flagging them from the queue itself. 

One of the ways they do this is through The digg community quickly sees through a lot of this a feature we have called Digg Spy. It allows you to see what people are digging, commenting on and burying - in real time. So there's a community of users that will sit there and select which modifiers to monitor. For example they may choose to see only those stories being buried and how they're being buried (as spam, as duplicate post, etc). And they'll actually work together as a community to filter out these types of stories that are being posted.

Richard: Do the algorithms you use in digg detect spam automatically - or is it all human intervention?

Kevin: There are systems behind the scenes that notify our moderators, who are watching the site, whether there's any activity that we can detect. It's probably 95% user-driven There are a few different methods that I really can't get into, that we have set up to keep an eye on things. We can tell based on what the average of incoming stories is, stories with certain types of links, and we basically look at all different types of things across the site to get a good index of what's going on - there are certain flags that are set up that notify us of certain types of fraudulent activity. 

But for the most part I'd say it's probably 95% user-driven, so the users handle most of that for us - which is really nice from the admin side, in that we don't really have to have a large staff that's policing the site. It's actually only one person who watches the site in general - it's not really that big a task because the community handles most of it.

On Personalization

Richard: Onto the plans for personalization, I read a fantastic article in BusinessWeek at the end of last year in which you said that you're hoping to expand digg into more of a personalized system in which people can create their own communities, e.g. for non-technical topics. How's that coming along, when will we see the first signs of that?

Kevin: Well that's something we're working on right now. One of digg's top priorities is to expand outside of tech news It's one of our top priorities to expand outside of just technology-related news. We want to open digg up to just about anyone and everyone that wants to express their interest in any type of news story or Web content. So we're in the process of working on that right now and we hope to introduce this in the next few months.

Richard: Just before I got on Skype with you, I sent an email out to the Web 2.0 Workgroup (which I'm a founding member of) and asked if they had any questions they'd like to ask. One of them is from Dion Almear from ajaxian.com, who asked: "At the moment digg doesn't work for me as I am not a 17 yr old Xbox playing PHP hacker." He's obviously being a bit generic there, but will there be a variety of social networks in digg in the future? It's kind of different to the personalization question I asked earlier, because this is about whether there will be more formal digg social networking communities that you'd set up?

Kevin: Right now the features that we have for connecting users are primarily digg will enable users to create cliques of users the friends functionality that allow users to monitor what friends are digging, commenting on, and submitting. But we're very much focused on expanding and allowing users to create cliques of users that get along and want to discuss things on digg. So if there's any type of content, whether it be news or no matter what we get into later on - we want digg to be the location where users can come and share that information easily.

APIs

Richard: John Musser from programmableweb.com wants to know about your plans for an API. I think you mentioned that in the BusinessWeek article, so what's the latest on that?

Kevin: Shortly after we roll out into different areas outside of tech, we'll be finalizing the spec for the API and releasing it - and pretty much giving anyone who wants to create an application to manipulate the digg data the ability to do so. Hopefully we'll see this out right after we launch the new sections of digg.