Team Ninja Bulletin Board  

Go Back   Team Ninja Bulletin Board > DC Vault > DC Vault Problems and Suggestions

Reply
 
Thread Tools Display Modes
  #1  
Old 8th January 2008, 05:07 PM
Accs Accs is offline
Ninja Recruit
 
Join Date: Jan 2007
Location: San Diego, CA
Posts: 26
Lightbulb Request - Formal procedures for introducing new projects to the DC-Vault

Moderators - If this thread is unacceptable in this forum, please inform me (email) and remove it.

I would like to know what the formal procedures are for:
  1. Adding a new project to the DC-Vault. There are some general guidelines in the the DC-Vault FAQ, but these are VERY general, and I think they could stand to be cleaned up a bit.
  2. Archiving a completed project from the DC-Vault.
  3. Removing a non-compliant project from the DC-Vault.
If there are no formal procedures for these, I would like to request that such procedures be created. Finally, I would like to open a path where the users can request changed to the above procedures, for the improvement of the process.

The reason for this request is that I have seen a number of anomolies in some of the recently added projects, that I don't believe should be there. I believe that these anomolies are contrary to the long-term goals of the DC-Vault.

For those making suggestions, please be CONSTRUCTIVE. A comment of "xxx sucks" would be inappropriate. Comments of "xxx could be made better by ...", or "I think we need to improve xxx" would probably be acceptable.
Reply With Quote
  #2  
Old 8th January 2008, 08:38 PM
Rusty Rusty is offline
Owner
 
Join Date: Jan 2004
Location: Tasmania, Australia
Posts: 11,368
I dont think I quite understand what you want.. Maybe if you could start by offering some suggestions..

A project can be added to the Vault once it meets the guide laid out in the FAQ
__________________
RUSTY


Team Ninja Forever : Once a Ninja, always a Ninja - Team Ninja

"I'm a SAS NINJA"

Drafted to the SAS
Dump of the week in Folding@Home   Dump of the month in Folding@Home   Stomp of the week in Folding@Home   Stomp of the month in Folding@Home  
Reply With Quote
  #3  
Old 9th January 2008, 02:47 AM
umccullough umccullough is offline
Team Haiku Member
 
Join Date: Dec 2006
Location: Grass Valley, California, USA
Posts: 131
Oblig. Disclaimer: I'm not an administrator or decision maker for DC Vault, just a lowly team "captain" of sorts.

I think having some additional qualifications for projects acceptable for the Vault would be a good idea.

There's clearly been a large upswing in the number of DC projects in the last year or two (mostly BOINC) - which presents a challenge. It's hard to determine the quality of a project, it's administration, or the results without some amount of careful scrutiny and/or a review period. I think there should be some additional parameters around plentiful work queues, total project scope and estimate time to completion (it may not be worth it to bring <12 month projects into the Vault).

I sorta like the idea of "archiving" projects - or optionally viewing team positions including all archived projects... but I'm not sure if it really meets the goals of The Vault. Right now they appear to simply be removed - probably because they fail to meet one of the criteria (such as no longer allowing registration).

However...

I get the impression that DC Vault is not just a stats aggregator, but rather a continual challenge that keeps the competitive teams involved with the newer, smaller projects rather than concentrating on domination of the longer running, larger ones. This means the removal of "extinct" projects entirely sort of makes more sense.

For the smaller teams (like Team Haiku), it can be somewhat challenging to figure out the best way to allocate our resources across ALL of the Vault projects in order to get the best "bang for the buck" on Vault ranking... which isn't necessarily a bad thing. We don't have the advantage of being around since the "beginning of time" when there were just a few projects worth crunching for. It's even worse when those projects are gone, dead, or no longer provide work.

So, back to the procedures for removing projects from the vault:

What exactly ARE the thresholds that a project must pass before they qualify for removal? I can remember some projects that sat dormant on the Vault forever before they got removed.
Reply With Quote
  #4  
Old 9th January 2008, 03:50 AM
Accs Accs is offline
Ninja Recruit
 
Join Date: Jan 2007
Location: San Diego, CA
Posts: 26
Quote:
Originally Posted by russkris View Post
I dont think I quite understand what you want.. Maybe if you could start by offering some suggestions..
You have official criteria (somewhat loose, but official criteria nonetheless) for adding a project to the DC-Vault. I'm interested in seeing similar criteria for archiving/removing a completed project, and for removing a project if it becomes, or is found to be, non-compliant with the admittance criteria.
Quote:
A project can be added to the Vault once it meets the guide laid out in the FAQ
I'm not sure that the guidelines in the FAQ are adequate. Examples of this are as follows:
  1. APS@home - This project has had little work since it was put into the DC-Vault. I don't know where the problem is, but either the Vault participants ripped through the work faster than expected, or the project is unable to provide work on a reasonable basis. In either case, it's currently borderline on the "active" part of it's requirement (from the FAQ). I believe that a "Beta" period should exist, where it is in the vault, and scored, but the scores are not counted. This will:
    • Let the teams know that the project is soon to be placed into the DC-Vault
    • Allow the teams to debug the project on a large scale
    • Allow the teams to report problems with the project to the project manager, and/or the DC-Vault, as appropriate
    • Allow the teams to manage their resources to handle the additional project
  2. Yoyo@home - This is basicly a BOINC project wrapper for non-BOINC projects. All work completed on yoyo@home goes to a single "yoyo@home" account in the projects that it front-ends. This work does NOT credit your work/points or even the Team, in the project being run. The issue here is that someone owns that account, and they could easily add it to a team, creating a HUGE bump in their team's standings in the projects. A BOINC wrapper for a non-BOINC project is fine. Not having that wrapper forward CORRECT information on the user completing the work to the non-BOINC project is potentially bad, and should be adequate reason to exclude the project from the DC-Vault. Using the project's "anonymous" account would be fine, if there was a way to verify that it wasn't changed (either deliberately or by accident).
  3. Radio Network Design - This was (is?) on a list of possible projects to be added to the DC-Vault. It appears to me that this is a project whose primary beneficiary is the Mobile Phone Service Providers, in that it reduces their cost to achieve a given level of coverage. I consider this to be a commercial project with the financial gain limited to a VERY few (the previously mentioned providers). Because of this, I don't think it should be eligable to be included as a DC-Vault project. MoneyBee is different in that anyone who works on the project has access to the same information.

In addition, as previously stated, there are NO official criteria for archiving/removing a project when itís complete, nor is there any such criterion for removing a project that becomes, or is found to be, in violation of the acceptance criteria.
Reply With Quote
  #5  
Old 12th January 2008, 04:15 PM
Accs Accs is offline
Ninja Recruit
 
Join Date: Jan 2007
Location: San Diego, CA
Posts: 26
It's been over three days, and no response of any sort has been provided to either umccullough or myself. No "good idea", no "we'll look into it", no "waiting for more input", not even a "we don't like it, get lost". Is there any chance of making this a two-way exchange?

The initial response from russkris indicated that an open discussion was possible, but a discussion requires responses. Is anyone listening?
Reply With Quote
  #6  
Old 13th January 2008, 01:29 AM
Nanobot Nanobot is offline
Owner
 
Join Date: Dec 2001
Location: England
Posts: 26,549
We have been waiting for more feedback from the general community but this has not happened.

The rules for adding projects is already documented and any active project which no longer adheres to these rules will be removed. If you think the rules should be changed then a new thread should be started to discuss proposed changes laying down your proposed changes.

As for archiving, there is no plan to introduce this as I do not have the time to think about the design or implications of it at the moment.

Any project which is available to the public which meets the criteria for inclusion will be accepted. We are not going to start policing the worth of a project e.g. Radio Network Design as that would cause untold discussion as to what was a worthwhile project. If individuals or teams do not want to participate in a project then that is fine but we will not exclude any project which does not meet the project inclusion rules. This includes projects issuing old work as teams can get credit for processing those units.

The yoyo problem you mention can be adressed by posting a thread asking us to look into a team where you think this has happened after you have checked the teams project participation. If we feel this is an issue we can always remove the project from that team.

umccullough mentioned the problem of the small teams so I will give a little hint here; why do we list the number of Vault teams participating in a project on the Projects page

Personal note here, not as a Vault admin or Team Ninja admin, the use of formal procedure in the thread title causes a slight concern as once formal procedures are introduced you usually need a large team of people to police those procedures, over 20 years of IT experience has taught me this. So I do not think that formal procedures will be introduced into the Vault which is just a bit of fun

And finally [groans of thank goodness from the reading populace] the reason we had a period of inactive projects not being removed was that we were all busy and then I needed to update the system so that projects could be temporarily removed.

Remember

Rule 1 have fun

Rule 2 see rule 1

__________________
Nanobot image courtesy of Tim Fonseca
Reply With Quote
  #7  
Old 13th January 2008, 05:27 AM
cswchan's Avatar
cswchan cswchan is offline
Crunchers Inc
 
Join Date: Mar 2006
Location: Mississauga, Ontario, Canada
Posts: 191
Quote:
Originally Posted by Nanobot View Post

Remember

Rule 1 have fun

Rule 2 see rule 1

Very well put Nano...

__________________
Crunchers Inc
Reply With Quote
  #8  
Old 13th January 2008, 07:46 AM
Accs Accs is offline
Ninja Recruit
 
Join Date: Jan 2007
Location: San Diego, CA
Posts: 26
Quote:
Originally Posted by Nanobot View Post
We have been waiting for more feedback from the general community but this has not happened.
And the community has been waiting to see if this turned into a discussion. The tone of your response has shut that path down. So much for being willing to accept suggestions.
Reply With Quote
  #9  
Old 13th January 2008, 05:08 PM
Da-Grizz Da-Grizz is offline
Ninja Jotohei
 
Join Date: Sep 2002
Location: KWSN off (*) course
Posts: 58
Ni !

This discussion SHOULD continue - ACCS and UMCCULLOUGH have some VERY valid points , and I agree with most if not all of them .
Yes one point is to have fun in the stats , but not at the expense of blindly crunching for points in dubious projects , and some projects are dubious , tapping into the Boinc community for their own gain .
Most serious "Crunchers" are able to review and make up their own minds if a project is worthwhile , but many cannot and look to concepts such as the DC Vault for guidance . Inclusion of a project in the Vault constitutes such guidance whether you agree or not . Look at the mad scramble for points in the recently included projects in the Vault to see what I mean . As such it caries responsibility , and maybe the inclusion rules need to be revised ?
For example , state whether a project is indeed a disguised private venture , (in your opinion) designed to harvest the cycles of the DC community (and they ARE out there) for their own (fiscal) gain .

Off of soap box now , and please do not construe this missive as a negative against DC Vault , I assure it is not , your Vault has done wonders in getting people to contribute their CPU cycles (for free) in worthwhile projects .

Set up a proper discussion forea , and let people know what projects you are considering for inclusion on a proactive baseis , not the fait-acomple as at present . Maybe some of us will give you a lot more feed-back then , and shut up moaning

Best Regs KWSN Da-Grizz

Last edited by Da-Grizz; 13th January 2008 at 05:19 PM. Reason: missed a name
Reply With Quote
  #10  
Old 13th January 2008, 08:24 PM
Rusty Rusty is offline
Owner
 
Join Date: Jan 2004
Location: Tasmania, Australia
Posts: 11,368
Quote:
Set up a proper discussion forea , and let people know what projects you are considering for inclusion on a proactive baseis , not the fait-acomple as at present . Maybe some of us will give you a lot more feed-back then , and shut up moaning
Thats a great idea, but what I use to do is post in the new project forum and ask people to give me feedback about each project, but I very rarely got any info back, and it turns out the same people would reply time and time again..

This is why we are taken back by the sudden interest in the Vaults rules...

I wish we had people like yourself helping with the project inclusion threads..

Please feel free to write your own and post it everyone can read and input more or less
Reply With Quote
  #11  
Old 14th January 2008, 02:11 AM
umccullough umccullough is offline
Team Haiku Member
 
Join Date: Dec 2006
Location: Grass Valley, California, USA
Posts: 131
I think we can still keep the discussion moving in a positive direction

I got from Nanobot that he's not interested in setting up any method of tracking "Archived Projects". To be clear, I wasn't exactly for or against the idea of Archived Projects myself - in fact, I find them to be a problem for newer teams to overcome (Free-DC's DCR system is a good example of this problem).

As for making the rules for project inclusion more stringent - I think russkris is correct: Not enough people get involved.

Even if no hard rules are set, I'd like to see the Vault start scrutinizing the following before adding new projects:

* "Solid" project administration
* Plentiful work queue
* Estimated time to complete project, and whether work will be available after that
* and to some degree: Usefulness of the project

I think these can't be necessarily quantified in all cases, but if those questions are at least asked, and vault participants can attempt to provide some reasonable answers to those questions (possibly by asking the project admins themselves) - it might help the rest of us determine the worthiness of the project.

The final point above may or may not serious impact whether the project gets accepted into the vault, but if the project's acceptance is teetering - the project's usefulness could help with a final decision. There are clearly some projects out there of questionable usefulness, fortunately few, if any, have graced the Vault's project list yet

If a project aims to provide personal financial gain for the project admins - it would be nice to know that before inclusion into the vault. However, it's the responsibility of persons who have this information to come forward and mention it so that it can be investigated further.
Reply With Quote
  #12  
Old 14th January 2008, 03:37 AM
Accs Accs is offline
Ninja Recruit
 
Join Date: Jan 2007
Location: San Diego, CA
Posts: 26
Quote:
Originally Posted by russkris View Post
we are taken back by the sudden interest in the Vaults rules
It's not a sudden interest. I've seen projects with various issues come and go in the vault, and mostly the problems weren't enough to trigger a response. The YoYo project nudged me to the point where I felt that I had to ask the above questions.

To clarify, when I wrote "official", I meant nothing more than a set of qualifications, similar to the qualifications you have for vaulting a project. It appears that your qualifications for removing a project are the same.

Additionally, I said "archival", meaning that the project would be considered to be completed, and would be removed from scoring. It would be nice to keep a copy of the final project scores, but that's not necessary.

I'd still like to see a few modifications to the vaulting requirements.
Reply With Quote
  #13  
Old 17th January 2008, 01:40 AM
Rusty Rusty is offline
Owner
 
Join Date: Jan 2004
Location: Tasmania, Australia
Posts: 11,368
Ok... I see your points, and all of them...

The reason to keep the rules as general as they are, is so that basically every project has a chance of being added..

There is going to be no archiving of projects in the Vault.. Pure and simple, it a game, a bit of fun.. Or else it will turn in to another stats site and there are plenty of them..

As to projects like yoyo@home, well plain and simple, wrapper projects are totally new to me and I have not read much about them. But what I understand is yoyo gets the points in the project it has wrapped eg: DNET.. then a user crunching for yoyo@home gets BOINC points..

So given that, yoyo at home wouldn't be create a DC-Vault team, that would be just un-fair
Reply With Quote
  #14  
Old 17th January 2008, 07:59 PM
Accs Accs is offline
Ninja Recruit
 
Join Date: Jan 2007
Location: San Diego, CA
Posts: 26
Quote:
Originally Posted by russkris View Post
There is going to be no archiving of projects in the Vault.. Pure and simple, it a game, a bit of fun.. Or else it will turn in to another stats site and there are plenty of them..
OK.
Quote:
As to projects like yoyo@home, well plain and simple, wrapper projects are totally new to me and I have not read much about them. But what I understand is yoyo gets the points in the project it has wrapped eg: DNET.. then a user crunching for yoyo@home gets BOINC points..

So given that, yoyo at home wouldn't be create a DC-Vault team, that would be just un-fair
That's one of the problems with yoyo. The points go to a specific "user" account. There exists the possibility for that user account to be joined to a team. The DC-Vault isn't in charge of the user, and probably wouldn't want to be.

Perhaps an additional set of rules for "BOINC wrapper" projects is needed. One of these rules might be that the wrapper place the points into the project's "anonymous" account.

Another might be that a BOINC project not be part of a BOINC wrapper. It makes no sense to go through two BOINC layers. Am I missing something here?

Personally, I don't see a reason for ANY wrapper project to be in the vault. Creating a BOINC wrapper for a non-BOINC project is fine, but placing it into the vault as a separate project? NO. If the wrapper passes the user information to the wrapped project, then everything works correctly, without the additional overhead of another project.

Not allowing wrapper projects in also significantly reduces the risk of abuse (one of my problems with yoyo). I'm not saying that this abuse will occur (in fact, I doubt it will), but it IS a risk, and it's a risk that you don't have to take.

Finally, when a "dumb" wrapper is used, the underlying project shows points for the wrapper. The user/team that performed the work isn't given ANY credit on the project web page. Of course, if the team receives vault points for the wrapper project, they shouldn't also receive vault points for the wrapped project.

The point here is:
  1. Make the wrapper project smart.
  2. Don't vault the smart wrapper.
  3. Let the user/team receive credit in the base project.
  4. The user/team will now receive BOINC points for their work.
Quote:
Ok... I see your points, and all of them...
Any feedback on the possibility of a 2-4 week "Beta" period for new projects? I listed several of the benefits previously.
Quote:
The reason to keep the rules as general as they are, is so that basically every project has a chance of being added..
Just because a project exists, doesn't mean that it should be in the vault. First, there are the issues with the wrapper projects. Before a wrapper project is considered, I believe that the underlying project(s) need to be looked at. I believe that it's MUCH better to give credit for the user/team on the actual project, than to give credit for the wrapper.

Second, as you wrote previously, "Pure and simple, it a game, a bit of fun". When purely commercial projects (like Radio Network Design) enter the vault, I believe that it ceases to be a game.
Reply With Quote
  #15  
Old 17th January 2008, 10:07 PM
Rusty Rusty is offline
Owner
 
Join Date: Jan 2004
Location: Tasmania, Australia
Posts: 11,368
Quote:
Originally Posted by Accs View Post
Second, as you wrote previously, "Pure and simple, it a game, a bit of fun". When purely commercial projects (like Radio Network Design) enter the vault, I believe that it ceases to be a game.
How is not a game if Radio Network Design enters the Vault... Please explain..


I think the from now on, when I or someone else posts about a project, I would like to see everyone contribute to the thread, including yourself, it seems you know far more then I about Distributed Computing then me..

I cant see the rules changing.. But I can see the discussion increasing prior to a projects addition... And on the same wave-length, when a project is to be taken out the same process should be adopted..

As far as yoyo@home goes, there was no comment about the things you are concerned about.
http://www.team-ninja.com/vbulletin/...ad.php?t=42194
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT. The time now is 05:33 AM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.