comparative-judgement

cj

Comparative Judgement Algorithms

If you would like to learn more about Comparative Judgement, or to see how it works in action, please visit: https://nomoremarking.com

Component Dependencies

• Underscorejs.org

Simulations

//Simulate a set of comparative judgement results
//Saved to the files out.csv and alpha.csv
node index.js -s -p 50 -j 1250 --selection 'adaptive' --thru 49 --ap 4 --iters 10 --seed 1234 --offset 0
// Run 10 simulations of 50 candidates receiving 1250 judgements (total) using the adaptive method
// With an expected number of 49 judgements per candidate
// With an acceleration parameter of 4
// And a seed for the simulation of 1234
// An offset for the adaptive of 0.5 logits

node index.js -s -p 50 -j 1250 --selection 'adaptive' --thru 49 --ap 4 --iters 10 --seed 1234 --offset 0 --rnd
//Repeat the above but with random judgements (noise)

node index.js -s -p 20 -j 190  --selection 'distributed' --iters 10 --seed 1234
// Run 10 simulations of 20 candidates receiving 190 judgements using the distributed method
// And a seed for the simulation of 1234

Pair Selection Methods

function selectionNonAdaptive (players)

1. Shuffles the items, then selects the item with fewest judgements
2. Pairs the item with any item not yet paired with
3. If all pairs exhausted, returns the item with the next fewest judgements

function selectionAdaptive( players, thr, AP, offset )

Uses the progressive method (Barrada, Olea, Ponsoda, and Abad, 2008, 2010; Revuelta and Ponsoda, 1998)

thr - the number of comparisons expected for a item (analagous to test length)
AP - the acceleration parameter
offset - offset the initial theta so the pair chosen is on average some distance away from the original

With the progressive method the item selected is the one that maximizes the sum of two elements,a random part and a part determined by the Fisher information. At the beginning of the test, the importance of the random element is maximum; as the test advances, the information increases its relevance in the item selection. The speed for the transition from purely random selection to purely information based selection is determined by the acceleration parameter, set by the argument AP, where higher values imply a greater importance of the random element during the test.

function selectionByJudge(idJudge, players, decisions)

Ensures every judge sees every pair. Useful if you want to construct a reliable scale for every judge.

Utility Functions

Return a scale for every judge from a csv download of decisions from nomoremarking.com

node index.js -b -d ~/Downloads/image-study-beauty.csv -e '[e-mail]','[e-mail]'

Functions

• rasch = function(ability, difficulty)
• estimateJudges = function(players, decisions, judges, callback)
• average = function(a)
• estimateReliability = function(players)
• markerInfit = function(players, decisions, judges, updateHolder, callback)
• resInfo = function(p)
• estimateCJ = function(task, decisions, players, iters, controller)
• cjEstimation = function(task, playerids, players , decisions, updateHolder, callback, iters, controller)
• btmModel = function(dat, fixTheta, maxIter, conv, eps, callback)

JSON Objects

Players

{
        "_id" : "22QCYH8oJYsoHYz69",
        "comparisons" : 0,
        "createdAt" : "2014-05-06 08:59:28",
        "decisions" : [  "tgHjzRze7jdTqbyYA" ],
        "observedScore" : 0,
        "owner" : "xDMzcHpZDhKXJ7wzk",
        "selected" : 0,
        "task" : "YCpw984EfTJmeTJeR",
        "timeTaken" : 0,
        "trueScore" : 0
}  

Judges

{
        "_id" : "23aa6mFLA5MaWBXPZ",
        "comparisons" : 20,
        "createdAt" : "2014-05-078 15:29:28",
        "email" : "a@b.c",  
        "frm_welcome_email_sent" : 0,
        "owner" : "avbgKt4ygLfyoWQGw",
        "quota" : 20,
        "task" : "6ZEKR54w45a5hn8gk",
        "timeTaken" : 11443.797999999999,
        "trueScore" : 0.8063503582501295  
}

Decisions

{
        "task" : "8M5b8yxRatHicgaqu",
        "chosen" : "hpGN6h4TJJrurRBrZ",
        "notChosen" : "zA3ZRBJnTHSrytaxo",
        "timeTaken" : 10.207,
        "judge" : "ZyNFixaAqxTADWmZw",
        "createdAt" : "2014-05-07 14:23:34",
        "_id" : "235x5zwkfm32DvpCZ"
}  

Contributing

The Grunt CLI needs to be installed (globally):

npm install -g grunt-cli

(Could be you need to do it as sudo.) Next cd to the project directory and:

npm install

This will install all necessary dependencies.

Building

Some parts of this project are client-side compatible. To build the distribution files:

grunt build

This will generate the distributable and minified files into the dist directory.

Testing

The tests use Mocha as the testing framework, must.js as an assertion library and sinon.js for mocking and spying.

To run all the tests:

npm test
# or 
grunt test

Benchmarking

Run all benchmarks:

grunt benchmark

Results are written to benchmarks/results.csv.