Team Ninja Bulletin Board  

Go Back   Team Ninja Bulletin Board > DC Vault > DC Vault Problems and Suggestions

Reply
 
Thread Tools Display Modes
  #1  
Old 13th November 2010, 04:00 PM
Beyond's Avatar
Beyond Beyond is offline
Ars Technica
 
Join Date: Jan 2007
Location: Rum River
Posts: 186
Rules for project inclusion

Current DC-Vault rules for project inclusion:
Quote:
Project eligibility is governed by the following guidelines:

The project must:
  • be active
  • accept new members and teams immediately upon registration
  • have parsable team stats
  • have team stats that are updated regularly
  • provide a client program which runs on a local PC
The project must not:
  • be a keylogger or mouseclick counter
  • have a maximum number of teams or members or exclude any country
I'd like people to kick around some ideas concerning new rules for Vault inclusion:

1) Projects must have 100 or more active users in the last 28 days.
2) Projects must have 10 or more active teams in the last 28 days.

The numbers above are simply suggestions off the top of my head. Maybe they're a bit high. These new rules (in some form) would make the Vault more vital and competitive by eliminating outdated and little used projects, thus allowing newer teams a CHANCE to actually do well. Older teams would still hold an advantage for their point accumulations in vital currently used projects.

Discussion?
Reply With Quote
  #2  
Old 13th November 2010, 10:21 PM
Xaverius's Avatar
Xaverius Xaverius is offline
Dutch Power Runner! Uhm, Cow
DPC - crew member
 
Join Date: Aug 2008
Location: Arnhem, the Netherlands
Posts: 98
Does this have any influence on the scores as we do have now in the vault?

It is normal that older teams participating in a project such as the DC-vault have a (big) lead, they've put enough effort in it to get that lead. But I also agree that a project must have some kind of a active user base to be in the vault.
Reply With Quote
  #3  
Old 14th November 2010, 01:15 AM
Beyond's Avatar
Beyond Beyond is offline
Ars Technica
 
Join Date: Jan 2007
Location: Rum River
Posts: 186
Quote:
Originally Posted by Xaverius View Post
Does this have any influence on the scores as we do have now in the vault?

It is normal that older teams participating in a project such as the DC-vault have a (big) lead, they've put enough effort in it to get that lead. But I also agree that a project must have some kind of a active user base to be in the vault.
It depends on where the trigger levels are set. I'm sure there are projects that wouldn't meet the 100/10 level. Maybe that's too high a level though? It just seems that if there are less than 100 (or 50, or 25) in the world running a project it can't be very relevant anymore.
Reply With Quote
  #4  
Old 15th November 2010, 10:09 AM
DigiK-oz DigiK-oz is offline
Dutch Power Cows
 
Join Date: Oct 2008
Location: netherlands
Posts: 108
I thought one of the reasons to start DC-Vault in the first place was to give some exposure to smaller projects? Ruling out projects for being too small seems to contradict this.

Having said that, I do agree that active participation could and maybe should be part of the inclusion/exclusion rules. Most projects that are just starting are usually ruled out anyway, because they have rough edges with their scoring or otherwise. Once new projects mature, they will usually be above some sort of lower limit. So, a rule like this will usually hit "older" projects in which interest has diminished.

Given that DC-Vault is a team comparison, I would rather not set a limit on the number of active users, but only on active teams. And maybe not a fixed limit like 10, but a soft limit like a percentage of the total number of participating teams in the project? Either limit could easily be checked by the statsrun, and a 14 day countdown posted on the forum. This might give teams an incentive to redirect attention to a project to safeguard it from being removed
__________________
DC-Vault statistics
Reply With Quote
  #5  
Old 16th November 2010, 10:31 AM
runesk's Avatar
runesk runesk is offline
Ninja Jotohei
 
Join Date: Apr 2009
Location: Trondheim
Posts: 52
Quote:
Originally Posted by DigiK-oz View Post
I thought one of the reasons to start DC-Vault in the first place was to give some exposure to smaller projects? Ruling out projects for being too small seems to contradict this.
I had the same thought.

I also would like to second opyrt's last question. What will the Vault benefit from such a change? I really can't see how the Vault will get any better from such a change in the rules.

.R
__________________
http://fuldans.se/?v=ftkudnpvvm
Reply With Quote
  #6  
Old 15th November 2010, 09:30 PM
Nanobot Nanobot is offline
Owner
 
Join Date: Dec 2001
Location: England
Posts: 22,178
I think this is a good idea as it could possibly help smalller projects, in that if they are starting to lose teams to other projects they may get renewed interest if they are in danger of being removed from the Vault.

Before the discussion goes any further I would like to rule out the user activity as the time required to download all of the user information, for all of the projects in the Vault, is time prohibitive.

I like the idea of a percentage but I think this could be used in conjunction with a minimum number of active teams, thereby catering for small and large projects. My initial though would be for 25 active teams or 10% of all teams, with a score.

Another colllection restriction is that WCG would be excluded from any checks as there is no way, that I know of, to determine the number of active teams.

If this is agreed then some patience will be required as we do not currently save team points because the Vault is position based. So quite major changes will be required of the stats collection routines.

Hopefully others will join in this discussion as I think it is a good enhancement to the Vault
__________________
Nanobot image courtesy of Tim Fonseca
Reply With Quote
  #7  
Old 16th November 2010, 10:24 AM
opyrt's Avatar
opyrt opyrt is offline
Ninja Gunso
 
Join Date: Feb 2008
Location: Oslo, Norway
Posts: 253
I'm struggeling to see any benefit from this change... Anyone care to enlighten me?
Reply With Quote
  #8  
Old 16th November 2010, 03:27 PM
DigiK-oz DigiK-oz is offline
Dutch Power Cows
 
Join Date: Oct 2008
Location: netherlands
Posts: 108
Quote:
Originally Posted by opyrt View Post
I'm struggeling to see any benefit from this change... Anyone care to enlighten me?
if it is not just a change in the inclusion rules, but also a possible reason for removal, it will prevent older projects, that almost everyone has lost interest in, to be dominating the Vault score. Like the original Eon, for instance, where not many teams participated, and also the scoring changed dramatically at some point. It had about 50 teams or so, giving about 200 points per position gained, and with the degraded scoring (100-fold and more) there was NO way for new(ish) teams to get anywhere near the top, no matter the number of PC's used. This caused the early adopters of the project to hold several thousand Vault-points advantage with no effort.

Things like this could cause teams to lose interest in the Vault, simply because there's no real incentive in adding power to certain projects to gain a few points if there are projects where hundreds of points are given for a single position gain, but with a grand canyon of difference in points to the next higher team.

There are ways to overcome this, and this topic suggests one of them. Another possibility would be to adjust the points range for the Vault (0-10000) for the smaller projects, i.e. if less than 100 teams compete, set the maximum score to (#teams * 100). Or, as I suggested before, calculate the Vault score over a team's top-30 projects (allowing a team to ignore 5 projects they dislike for whatever reason whilst still staying competitive in DC-Vault).
Reply With Quote
  #9  
Old 7th March 2011, 03:17 PM
Beyond's Avatar
Beyond Beyond is offline
Ars Technica
 
Join Date: Jan 2007
Location: Rum River
Posts: 186
Quote:
Originally Posted by Nanobot View Post
I think this is a good idea as it could possibly help smalller projects, in that if they are starting to lose teams to other projects they may get renewed interest if they are in danger of being removed from the Vault.

Before the discussion goes any further I would like to rule out the user activity as the time required to download all of the user information, for all of the projects in the Vault, is time prohibitive.
Nanobot, since starting this thread, Bok has graciously added the ability to easily gauge member participation over the last 28 days. In fact it's now even easier than than measuring team participation. Here's the pertinent link for a BOINC project:

http://stats.free-dc.org/stats.php?p...pi&sort=last28

And a non-BOINC project:

http://stats.free-dc.org/stats.php?p...ad&sort=last28

Quote:
I like the idea of a percentage but I think this could be used in conjunction with a minimum number of active teams, thereby catering for small and large projects. My initial though would be for 25 active teams or 10% of all teams, with a score.

Another colllection restriction is that WCG would be excluded from any checks as there is no way, that I know of, to determine the number of active teams.

Hopefully others will join in this discussion as I think it is a good enhancement to the Vault
WCG is really not a problem. It has a mammoth number of teams and as far as user participation, we can now easily see that there are currently in excess of 60,000 active users:

http://stats.free-dc.org/stats.php?p...8&offset=60001

I don't think it would take any scripting changes. Just as with other conditions of inclusion it would be up to a Vault user to point out the concern and then post a thread.

Thought that since inclusion rules are being kicked around we should revisit this given the new capabilities Bok has provided on the great Free-DC stats site.
Reply With Quote
  #10  
Old 16th November 2010, 06:52 PM
runesk's Avatar
runesk runesk is offline
Ninja Jotohei
 
Join Date: Apr 2009
Location: Trondheim
Posts: 52
I would rather propose to request a removal of such projects like the old eOn, if the scoring premises is not equal during a projects lifetime.

Bad scoring models, will always be a problem, independent of the number of participants (though lower score for the same amount of calculations probably will lead to fewer participants).

I think that by requiring this kind of minimum participant requirements, the vault will end up containing mostly BIONC projects that tend to grant cobbles at a higher level than the average score pr calculation. BOINC combined has already been ruined by such projects (but mostly by comparing CPU and GPU score in the same scale )

I agree in Nanobot's wish for a more vital and alive Vault, but I disagree on this propose, as it will probably lead to those small projects getting even less calculation power at their hands.

I can think of one project that will be a major pain to find the number of participants; GIMPS. But Nanobot already made clear that participants never will be the requirement, and active teams should not be too difficult, even for GIMPS

I like think the idea of changing the scoring algorithm might be a better solution to diminish these smaller projects impact of the overall Vault standings; as a 2nd place in, say Muon1 DPAD is drastically lower than a 100th place in Fondling@HOME (but, hey Folding has massive support, Muon1 has marginally support).

This ended up being a somewhat unstructured post, but as I'm a bit tired I have no energy left to try to restructure it

.R
Reply With Quote
  #11  
Old 20th December 2010, 10:08 PM
fractal fractal is offline
ARS Technica
 
Join Date: Sep 2007
Location: earth
Posts: 75
The old eOn was a bad example as people quickly latch onto the scoring change and ignore the main point you raised. LLR is probably a better example. It has been around for a long time with some teams on it from the beginning. There are very few teams on the project so each position is worth hundreds of vault points. It is very difficult for a new team to make an impact. The proposal as I understand it would allow the removal of old projects that still have work but not many participants.

The current points based on ranking can be considered a feature, a flaw or just the way things are. I initially thought it was bizarre that a bazillion terraflops or crunching on FAH got fewer vault points than a couple dozen terraflops on a brand new project. But, that is part of the flavor of the Vault as it, by its very nature, encourages people to contribute to the latest greatest new project. People who fold proteins for Stanford or peer around the stars trying to find ET for Berzerkely will do so whether the vault is around or not.
Reply With Quote
  #12  
Old 21st December 2010, 03:39 PM
DigiK-oz DigiK-oz is offline
Dutch Power Cows
 
Join Date: Oct 2008
Location: netherlands
Posts: 108
Adjusting the scoring is of course another way to get things more interesting. As I suggested before, calculating the total score over all but the lowest-score 5 (or 3, or whatever) projects PER TEAM would make things more interesting I think, especially in the top of the rankings (and yes, my team would probably benefit ). It would allow (new) teams to totally ignore those long-running projects if they wish to do so without ruining their total ranking, whereas it would still give the high-ranked teams in those projects their (well-deserved!) credits.
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT. The time now is 01:05 AM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.