Register
Login
Resources
Docs Blog Datasets Glossary Case Studies Tutorials & Webinars
Product
Data Engine LLMs Platform Enterprise
Pricing Explore
Connect to our Discord channel

tasks.rst 1.4 KB

You have to be logged in to leave a comment. Sign In
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
  1. .. role:: hidden
  2. :class: hidden-section
  3. .. module:: fairseq.tasks
  4. .. _Tasks:
  5. Tasks
  6. =====
  7. Tasks store dictionaries and provide helpers for loading/iterating over
  8. Datasets, initializing the Model/Criterion and calculating the loss.
  9. Tasks can be selected via the ``--task`` command-line argument. Once selected, a
  10. task may expose additional command-line arguments for further configuration.
  11. Example usage::
  12. # setup the task (e.g., load dictionaries)
  13. task = fairseq.tasks.setup_task(args)
  14. # build model and criterion
  15. model = task.build_model(args)
  16. criterion = task.build_criterion(args)
  17. # load datasets
  18. task.load_dataset('train')
  19. task.load_dataset('valid')
  20. # iterate over mini-batches of data
  21. batch_itr = task.get_batch_iterator(
  22. task.dataset('train'), max_tokens=4096,
  23. )
  24. for batch in batch_itr:
  25. # compute the loss
  26. loss, sample_size, logging_output = task.get_loss(
  27. model, criterion, batch,
  28. )
  29. loss.backward()
  30. Translation
  31. -----------
  32. .. autoclass:: fairseq.tasks.translation.TranslationTask
  33. .. _language modeling:
  34. Language Modeling
  35. -----------------
  36. .. autoclass:: fairseq.tasks.language_modeling.LanguageModelingTask
  37. Adding new tasks
  38. ----------------
  39. .. autofunction:: fairseq.tasks.register_task
  40. .. autoclass:: fairseq.tasks.FairseqTask
  41. :members:
  42. :undoc-members:
Tip!

Press p or to see the previous file or, n or to see the next file

Comments

Loading...