Are you sure you want to delete this access key?
Some of the Datahack 2019 challenges will be (at least partially) judged by leaderboard scores.
These leaderboards will be updated live as datahackers submit their results during the competition. You can use the leaderboard ranking to get an idea of who is in the lead.
However, the results in the leaderboard are not final!
When a hacker submits results, we calculate and save two different scores:
While the competition is still ongoing, only the public validation scores are visible. The ranking is determined by these public scores. Participants can't know what their secret test scores are.
When the competition ends, we will flip a switch which disables any further submissions, and reveals the secret test scores to the whole world. The leaderboard will switch to displaying scores and ranks by the secret test scores instead of the public validation scores, and the winners will be chosen according to this ranking.
Each team needs to pick a name for themselves, and use it for submissions. Here is example python code for submission to the Armis leaderboard, but you can do it in any other way that you like:
predictions = my_trained_model(test_data)
submitter = "My Awesome Team Name"
from urllib import request
import json
jsonStr = json.dumps({'submitter': submitter, 'predictions': predictions})
req = request.Request("http://https://leaderboard.datahack.org.il/armis/api/",
headers={'Content-Type': 'application/json'},
method='POST',
data=jsonStr.encode('utf-8'))
resp = request.urlopen(req)
print(json.load(resp))
Result format: {'member': 'My Awesome Team Name', 'rank': 1, 'score': 1.618}
The rank and score returned from a submission are public validation scores only!
They don't necessarily reflect your score on the final test set used to declare the winners. It's definitely possible to overfit the validation set, be ranked first in the leaderboard for the whole event, and suddenly drop to a much lower rank when the competition ends and the test scores are revealed.
So, avoid overfitting to the leaderboard scores.
We won't rank teams according to their best submission, only according to their last submission.
This means that as datahackers, it's up to you to decide what your best model is.
Make sure to submit your best model, the one you want to be judged by, just before the competition ends and the leaderboards are locked. We will let you know in advance before this happens, so you have time to make sure.
We recommend tracking your various models and their public validation scores (and any other scores you calculate for yourselves) in an experiment tracking system, to be able to make the best choice.
Press p or to see the previous file or, n or to see the next file
Are you sure you want to delete this access key?
Are you sure you want to delete this access key?