Embedded Files
Simple guide
Dear students,
this is a simple guide that helps you to avoid typical mistakes in student projects in the field of machine learning and gives structure for a possible collaboration. These are only additional tips. You should be familiar with the university's general guidelines for academic writing and {seminar, bachelor, master} thesis.
Supervision
I highly recommend a research protocol listing accomplishments, conclusions, successes, and failures. On the one hand, this will help you to refresh the insights you have gained in the past months and it is handy as a guide for the written part when the practical is done. On the other hand, it allows me to gain a quick overview and see successes and failures during the time until the next meeting. The preferred option is google docs to always have an up-to-date document on which several people can work and comment.
Please update your research protocol at least weekly.
For our regular meetings two possible modes are preferred:
a) The student asks for an appointment after significant steps have been taken or questions have arisen.
b) We agree on regular (e.g. biweekly) meetings.
Please update the research protocol before the meetings so that both sides can be optimally prepared for the meeting. If possible, send questions in advance.
Quick questions can always be asked via Skype or e-mail.
Experiments
Before conducting experiments, think about suitable measurements to track the success of your experiments (e.g. F1, UAR, CCC, mAP). These must be consistent across all experiments that have the same prediction type (e.g. classification or regression). Have a look at the latest literature to see what is common in the field (e.g. emotion prediction as a regression task is CCC, object recognition mAP). Please, discuss the selected quantitative and qualitative metrics with me before starting the experiments to avoid repeating them.
Keep in mind to analyse the label imbalance/ distribution and find appropriate counter-measures (loss weights, up-/downsampling etc.)!
If no data partitions are provided, create meaningful partitions and run some test models (e.g. see Baseline Models below). The metrics should behave similarly on development and test set. Otherwise, the partitions must be resampled.
Before you develop your own model, look for existing solutions (state of the art).
You generally create baseline models and then try to develop your own concept/ model and/or make more complex solutions in order to get a better result. Typically, you provide three types of baseline models:
By-chance level: What results do you achieve by chance? For example, a balanced binary problem gives you always 50% ACC by predicting one class only.
Classification by-chance baselines:
“stratified”: generates predictions by respecting the training set’s class distribution.
“most_frequent”: always predicts the most frequent label in the training set.
“prior”: always predicts the class that maximizes the class prior.
“uniform”: generates predictions uniformly at random.
“constant”: always predicts a constant label that is provided by the user.
Regression by-chance baselines:
“median”: always predicts the median of the training set
“quantile”: always predicts a specified quantile of the training set,provided with the quantile parameter.
Basic baseline model: How do (simple) machine learning models perform?
A basic baseline is the result of a very simplistic solution. It is expected that you achieve a better score than the basic baseline models.
These can be sklearn models (e.g. SVMs, decision trees) or simple neural networks e.g. LSTM, Stacked-LSTM, Feedforward etc.
State-of-the-art baseline: How do advanced models perform (e.g. deep learning)
Finally, if you are dealing with a specific domain of machine learning (such as NLP), then you will typically pick baselines that are current state-of-the-art(SoTA) approaches - since you will usually want to demonstrate that your approach does better than these. This could be e.g. not fine-tuned Bert model or implementations from latest conference papers
If you are not able to beat the last baseline, see if your model brings other benefits e.g. less parameters (neurons) or better qualitative results for your specific problem.
In general, you want your approach to outperform the baselines. For example, you would want your 75% accuracy to be higher than any baseline you have run on the same data.
Optimise the models on development only e.g. loss or metric! Always assume you do not see the test data at all.
For replication, store every ‘(best) model’ and the configuration you want to report later. Do not forget that also negative results are important!
All figures (e.g. loss graph) you export should be exported as ‘.pdf’ for later latex use. ‘.png’/’.jpg’ does not scale and becomes blurry.
Always export the y and y_true for devel/test of your best model. This enables you to recalculate measurements and (re)build new figures etc., e.g., in case you want to change the style later.
3. Code
The code must be submitted together with your written dissertation.
The code should be clean and reproducible. All functionality should be executable using arguments (python argparse) without the need to make changes to the code. This also helps you to build your code in a way that you are able to repeat experiments if you change your research questions or input data.
The medium of code sharing should be github. Gitlab is also possible. If you want to have a private repository and don't have a premium account, you can ask me to create one for you. If not familiar yet, here you can find more information about github introduction videos and a great simple guide!
Further
After literature research and data collection, it is time to a) define a certain problem you see and want to solve b) propose 2-3 research questions that help you to solve this problem c) sketch an outline of the proposed work (with a set of milestones). The research questions function as a guideline during the work-in-progress and help you to design appropriate experiments in order to (dis)prove your hypothesis. To deliver a strong work and score a high grade, you should develop your own approach and keep evaluation methodical. Your approach (based on the literature analysis) is the starting point that is fleshed out, combined or extended in our thesis. I am always happy to discuss the research questions with you.
Before you start writing your thesis, discuss our thesis outline with me. I can support you finding a well-defined structure, if wished.
Block the last 4-6 weeks for writing only. If you see flaws in your concept (e.g. missing experiments) use the time to re-run these in parallel using the arguments you developed before - avoid any additional concept development or coding!
The thesis(es) is/are your first project(s) for which you are (mostly) responsible on your own. The idea of this guide is to help you to deliver a high quality project result and avoid frustration on this way. Do not hesitate to contact me, if you have further questions.
Happy hacking!
Lukas
Google Sites
Report abuse