Register
Login
Resources
Docs Blog Datasets Glossary Case Studies Tutorials & Webinars
Product
Data Engine LLMs Platform Enterprise
Pricing Explore
Connect to our Discord channel
Resources  
Docs Blog Datasets Glossary Case Studies Tutorials & Webinars
Explore Pricing
Product  
Data Engine LLMs Platform Enterprise
Connect to Dagshub Discord channel
Register
Login

Deci-AI
/
super-gradients
connected to https://github.com/Deci-AI/super-gradients.git

Watch 4
Issues 62 Pull Requests 44 Discussions 0

user_guide.html 70 KB
Permalink History Raw

You have to be logged in to leave a comment. Sign In
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999 
  1. <!DOCTYPE html>
    2. 

  2. <html class="writer-html5" lang="en" >
    3. 

  3. <head>
    4. 

  4.   <meta charset="utf-8" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
    5. 

  5. 

  6.   <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    7. 

  7.   <title>What is SuperGradients? &mdash; SuperGradients 1.0 documentation</title>
    8. 

  8.       <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
    9. 

  9.       <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
    10. 

  10.       <link rel="stylesheet" href="_static/graphviz.css" type="text/css" />
    11. 

  11.   <!--[if lt IE 9]>
    12. 

  12.     <script src="_static/js/html5shiv.min.js"></script>
    13. 

  13.   <![endif]-->
    14. 

  14.   
    15. 

  15.         <script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>
    16. 

  16.         <script src="_static/jquery.js"></script>
    17. 

  17.         <script src="_static/underscore.js"></script>
    18. 

  18.         <script src="_static/doctools.js"></script>
    19. 

  19.     <script src="_static/js/theme.js"></script>
    20. 

  20.     <link rel="index" title="Index" href="genindex.html" />
    21. 

  21.     <link rel="search" title="Search" href="search.html" />
    22. 

  22.     <link rel="prev" title="Training package" href="super_gradients.training.html" /> 
    23. 

  23. </head>
    24. 

  24. 

  25. <body class="wy-body-for-nav"> 
    26. 

  26.   <div class="wy-grid-for-nav">
    27. 

  27.     <nav data-toggle="wy-nav-shift" class="wy-nav-side">
    28. 

  28.       <div class="wy-side-scroll">
    29. 

  29.         <div class="wy-side-nav-search" >
    30. 

  30.             <a href="index.html" class="icon icon-home"> SuperGradients
    31. 

  31.           </a>
    32. 

  32. <div role="search">
    33. 

  33.   <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
    34. 

  34.     <input type="text" name="q" placeholder="Search docs" />
    35. 

  35.     <input type="hidden" name="check_keywords" value="yes" />
    36. 

  36.     <input type="hidden" name="area" value="default" />
    37. 

  37.   </form>
    38. 

  38. </div>
    39. 

  39.         </div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
    40. 

  40.               <p class="caption"><span class="caption-text">Welcome To SuperGradients</span></p>
    41. 

  41. <ul>
    42. 

  42. <li class="toctree-l1"><a class="reference internal" href="welcome.html">SuperGradients</a></li>
    43. 

  43. </ul>
    44. 

  44. <p class="caption"><span class="caption-text">Technical Documentation</span></p>
    45. 

  45. <ul>
    46. 

  46. <li class="toctree-l1"><a class="reference internal" href="super_gradients.common.html">Common package</a></li>
    47. 

  47. <li class="toctree-l1"><a class="reference internal" href="super_gradients.training.html">Training package</a></li>
    48. 

  48. </ul>
    49. 

  49. <p class="caption"><span class="caption-text">User Guide</span></p>
    50. 

  50. <ul class="current">
    51. 

  51. <li class="toctree-l1 current"><a class="current reference internal" href="#">What is SuperGradients?</a></li>
    52. 

  52. <li class="toctree-l1"><a class="reference internal" href="#introducing-the-supergradients-library">Introducing the SuperGradients library</a></li>
    53. 

  53. <li class="toctree-l1"><a class="reference internal" href="#installation">Installation</a></li>
    54. 

  54. <li class="toctree-l1"><a class="reference internal" href="#integrating-your-training-code-complete-walkthrough">Integrating Your Training Code - Complete Walkthrough</a><ul>
    55. 

  55. <li class="toctree-l2"><a class="reference internal" href="#integrating-your-training-code-main-components">Integrating Your Training Code: Main components:</a></li>
    56. 

  56. <li class="toctree-l2"><a class="reference internal" href="#integrating-your-training-code-complete-walkthrough-dataset">Integrating Your Training Code - Complete Walkthrough: Dataset</a></li>
    57. 

  57. <li class="toctree-l2"><a class="reference internal" href="#integrating-your-training-code-complete-walkthrough-model">Integrating Your Training Code - Complete Walkthrough: Model</a></li>
    58. 

  58. <li class="toctree-l2"><a class="reference internal" href="#integrating-your-training-code-complete-walkthrough-loss-function">Integrating Your Training Code - Complete Walkthrough: Loss Function</a></li>
    59. 

  59. <li class="toctree-l2"><a class="reference internal" href="#integrating-your-training-code-complete-walkthrough-metrics">Integrating Your Training Code - Complete Walkthrough: Metrics</a></li>
    60. 

  60. <li class="toctree-l2"><a class="reference internal" href="#integrating-your-training-code-complete-walkthrough-training-script">Integrating Your Training Code- Complete Walkthrough: Training script</a><ul>
    61. 

  61. <li class="toctree-l3"><a class="reference internal" href="#training-parameter-notes">Training Parameter Notes:</a></li>
    62. 

  62. </ul>
    63. 

  63. </li>
    64. 

  64. </ul>
    65. 

  65. </li>
    66. 

  66. <li class="toctree-l1"><a class="reference internal" href="#training-parameters">Training Parameters</a></li>
    67. 

  67. <li class="toctree-l1"><a class="reference internal" href="#logs-and-checkpoints">Logs and Checkpoints</a></li>
    68. 

  68. <li class="toctree-l1"><a class="reference internal" href="#dataset-parameters">Dataset Parameters</a></li>
    69. 

  69. <li class="toctree-l1"><a class="reference internal" href="#network-architectures">Network Architectures</a></li>
    70. 

  70. <li class="toctree-l1"><a class="reference internal" href="#pretrained-models">Pretrained Models</a></li>
    71. 

  71. <li class="toctree-l1"><a class="reference internal" href="#how-to-reproduce-our-training-recipes">How To Reproduce Our Training Recipes</a></li>
    72. 

  72. <li class="toctree-l1"><a class="reference internal" href="#supergradients-faq">SuperGradients FAQ</a><ul>
    73. 

  73. <li class="toctree-l2"><a class="reference internal" href="#what-type-of-tasks-does-the-supergradients-support">What Type of Tasks Does the SuperGradients Support?</a></li>
    74. 

  74. </ul>
    75. 

  75. </li>
    76. 

  76. </ul>
    77. 

  77. 

  78.         </div>
    79. 

  79.       </div>
    80. 

  80.     </nav>
    81. 

  81. 

  82.     <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
    83. 

  83.           <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
    84. 

  84.           <a href="index.html">SuperGradients</a>
    85. 

  85.       </nav>
    86. 

  86. 

  87.       <div class="wy-nav-content">
    88. 

  88.         <div class="rst-content">
    89. 

  89.           <div role="navigation" aria-label="Page navigation">
    90. 

  90.   <ul class="wy-breadcrumbs">
    91. 

  91.       <li><a href="index.html" class="icon icon-home"></a> &raquo;</li>
    92. 

  92.       <li>What is SuperGradients?</li>
    93. 

  93.       <li class="wy-breadcrumbs-aside">
    94. 

  94.             <a href="_sources/user_guide.md.txt" rel="nofollow"> View page source</a>
    95. 

  95.       </li>
    96. 

  96.   </ul>
    97. 

  97.   <hr/>
    98. 

  98. </div>
    99. 

  99.           <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
    100. 

  100.            <div itemprop="articleBody">
    101. 

  101.              
    102. 

  102.   <p><strong>Contact Information</strong></p>
    103. 

  103. <p>Email – <a class="reference external" href="mailto:info&#37;&#52;&#48;deci&#46;ai">support<span>&#64;</span>deci<span>&#46;</span>ai</a></p>
    104. 

  104. <p>**Israel <br />
    105. 

  105. **Sasson Hugi Tower, Abba Hillel Silver Rd 12, <br />
    106. 

  106. Ramat Gan, Israel</p>
    107. 

  107. <p><strong>Revision History</strong></p>
    108. 

  108. <table>
    109. 

  109.   <tr>
    110. 

  110.    <td>1.0.1
    111. 

  111.    </td>
    112. 

  112.    <td>December 2021
    113. 

  113.    </td>
    114. 

  114.    <td>Initial version
    115. 

  115.    </td>
    116. 

  116.   </tr>
    117. 

  117. </table>
    118. 

  118. <section id="what-is-supergradients">
    119. 

  119. <h1>What is SuperGradients?<a class="headerlink" href="#what-is-supergradients" title="Permalink to this headline"></a></h1>
    120. 

  120. <p>The SuperGradients PyTorch-based training library provides a quick, simple and free open-source platform in which you can train your models using state of the art techniques.</p>
    121. 

  121. <p>Who can use SuperGradients:</p>
    122. 

  122. <ul class="simple">
    123. 

  123. <li><p>**Open Source Users – **The SuperGradients can be used to easily train your models regardless of whether you ever have or ever will use the <span style="text-decoration:underline;">Deci platform</span>.</p></li>
    124. 

  124. <li><p>**Deci Customers – **The SuperGradients library can reproduce the training procedure performed by Deci for their optimized models.</p></li>
    125. 

  125. </ul>
    126. 

  126. </section>
    127. 

  127. <section id="introducing-the-supergradients-library">
    128. 

  128. <h1>Introducing the SuperGradients library<a class="headerlink" href="#introducing-the-supergradients-library" title="Permalink to this headline"></a></h1>
    129. 

  129. <p>The <strong>SuperGradients</strong> training library** **provides all of the scripts, example code and configurations required to demonstrate how to train your model on a dataset and to enable you to do it by yourself.</p>
    130. 

  130. <p>SuperGradients comes as an easily installed Python package (pip install) that you can integrate into your code base in order to train your models.</p>
    131. 

  131. </section>
    132. 

  132. <section id="installation">
    133. 

  133. <h1>Installation<a class="headerlink" href="#installation" title="Permalink to this headline"></a></h1>
    134. 

  134. <ul class="simple">
    135. 

  135. <li></li>
    136. 

  136. </ul>
    137. 

  137. <p><strong>To install the SuperGradients library –</strong></p>
    138. 

  138. <ol class="arabic">
    139. 

  139. <li><p>Run the following command on your machine’s terminal –</p>
    140. 

  140. <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">pip</span> <span class="n">install</span> <span class="n">super_gradients</span>
    141. 

  141. </pre></div>
    142. 

  142. </div>
    143. 

  143. </li>
    144. 

  144. </ol>
    145. 

  145. </section>
    146. 

  146. <section id="integrating-your-training-code-complete-walkthrough">
    147. 

  147. <h1>Integrating Your Training Code - Complete Walkthrough<a class="headerlink" href="#integrating-your-training-code-complete-walkthrough" title="Permalink to this headline"></a></h1>
    148. 

  148. <p>Whether you are a Deci customer, or an open source SuperGradients user- it is likely that you already have your own training script, model, loss function implementation etc.</p>
    149. 

  149. <p>In this section we present the modifications needed in order to launch your training using SuperGradients.</p>
    150. 

  150. <section id="integrating-your-training-code-main-components">
    151. 

  151. <h2>Integrating Your Training Code: Main components:<a class="headerlink" href="#integrating-your-training-code-main-components" title="Permalink to this headline"></a></h2>
    152. 

  152. <p><span style="text-decoration:underline;">SgModel </span>- the main class in charge of training, testing, logging and basically everything that has to do with the execution of training code.</p>
    153. 

  153. <p><span style="text-decoration:underline;">DatasetInterface</span> - which is passed as an argument to the SgModel and wraps the training set, validation set and optionally a test set for the SgModel instance to work with accordingly.</p>
    154. 

  154. <p><span style="text-decoration:underline;">SgModel.net</span> -The network to be used for training/testing (of torch.nn.Module type).</p>
    155. 

  155. </section>
    156. 

  156. <section id="integrating-your-training-code-complete-walkthrough-dataset">
    157. 

  157. <h2>Integrating Your Training Code - Complete Walkthrough: Dataset<a class="headerlink" href="#integrating-your-training-code-complete-walkthrough-dataset" title="Permalink to this headline"></a></h2>
    158. 

  158. <p>The specified dataset interface class must inherit from <strong>super_gradients.training.datasets.dataset_interfaces.dataset_interface</strong>, which is where data augmentation and data loader configurations are defined.</p>
    159. 

  159. <p>For instance, a dataset interface for Cifar10:</p>
    160. 

  160. <div class="highlight-default notranslate"><div class="highlight"><pre><span></span>
    161. 

  161. <span class="kn">import</span> <span class="nn">torchvision.datasets</span> <span class="k">as</span> <span class="nn">datasets</span>
    162. 

  162. <span class="kn">import</span> <span class="nn">torchvision.transforms</span> <span class="k">as</span> <span class="nn">transforms</span>
    163. 

  163. <span class="kn">from</span> <span class="nn">super_gradients.training</span> <span class="kn">import</span> <span class="n">utils</span> <span class="k">as</span> <span class="n">core_utils</span>
    164. 

  164. <span class="kn">from</span> <span class="nn">super_gradients.training.datasets.dataset_interfaces</span> <span class="kn">import</span> <span class="n">DatasetInterface</span>
    165. 

  165. 

  166. 

  167. <span class="k">class</span> <span class="nc">UserDataset</span><span class="p">(</span><span class="n">DatasetInterface</span><span class="p">):</span>
    168. 

  168. 

  169.    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s2">&quot;cifar10&quot;</span><span class="p">,</span> <span class="n">dataset_params</span><span class="o">=</span><span class="p">{}):</span>
    170. 

  170.        <span class="nb">super</span><span class="p">(</span><span class="n">UserDataset</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">dataset_params</span><span class="p">)</span>
    171. 

  171.        <span class="bp">self</span><span class="o">.</span><span class="n">dataset_name</span> <span class="o">=</span> <span class="n">name</span>
    172. 

  172.        <span class="bp">self</span><span class="o">.</span><span class="n">lib_dataset_params</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&#39;mean&#39;</span><span class="p">:</span> <span class="p">(</span><span class="mf">0.4914</span><span class="p">,</span> <span class="mf">0.4822</span><span class="p">,</span> <span class="mf">0.4465</span><span class="p">),</span> <span class="s1">&#39;std&#39;</span><span class="p">:</span> <span class="p">(</span><span class="mf">0.2023</span><span class="p">,</span> <span class="mf">0.1994</span><span class="p">,</span> <span class="mf">0.2010</span><span class="p">)}</span>
    173. 

  173. 

  174.        <span class="n">crop_size</span> <span class="o">=</span> <span class="n">core_utils</span><span class="o">.</span><span class="n">get_param</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">dataset_params</span><span class="p">,</span> <span class="s1">&#39;crop_size&#39;</span><span class="p">,</span> <span class="n">default_val</span><span class="o">=</span><span class="mi">32</span><span class="p">)</span>
    175. 

  175. 

  176.        <span class="n">transform_train</span> <span class="o">=</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Compose</span><span class="p">([</span>
    177. 

  177.            <span class="n">transforms</span><span class="o">.</span><span class="n">RandomCrop</span><span class="p">(</span><span class="n">crop_size</span><span class="p">,</span> <span class="n">padding</span><span class="o">=</span><span class="mi">4</span><span class="p">),</span>
    178. 

  178.            <span class="n">transforms</span><span class="o">.</span><span class="n">RandomHorizontalFlip</span><span class="p">(),</span>
    179. 

  179.            <span class="n">transforms</span><span class="o">.</span><span class="n">ToTensor</span><span class="p">(),</span>
    180. 

  180.            <span class="n">transforms</span><span class="o">.</span><span class="n">Normalize</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">lib_dataset_params</span><span class="p">[</span><span class="s1">&#39;mean&#39;</span><span class="p">],</span> <span class="bp">self</span><span class="o">.</span><span class="n">lib_dataset_params</span><span class="p">[</span><span class="s1">&#39;std&#39;</span><span class="p">]),</span>
    181. 

  181.        <span class="p">])</span>
    182. 

  182. 

  183.        <span class="n">transform_val</span> <span class="o">=</span> <span class="n">transforms</span><span class="o">.</span><span class="n">Compose</span><span class="p">([</span>
    184. 

  184.            <span class="n">transforms</span><span class="o">.</span><span class="n">ToTensor</span><span class="p">(),</span>
    185. 

  185.            <span class="n">transforms</span><span class="o">.</span><span class="n">Normalize</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">lib_dataset_params</span><span class="p">[</span><span class="s1">&#39;mean&#39;</span><span class="p">],</span> <span class="bp">self</span><span class="o">.</span><span class="n">lib_dataset_params</span><span class="p">[</span><span class="s1">&#39;std&#39;</span><span class="p">]),</span>
    186. 

  186.        <span class="p">])</span>
    187. 

  187. 

  188.        <span class="bp">self</span><span class="o">.</span><span class="n">trainset</span> <span class="o">=</span> <span class="n">datasets</span><span class="o">.</span><span class="n">CIFAR10</span><span class="p">(</span><span class="n">root</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">dataset_params</span><span class="o">.</span><span class="n">dataset_dir</span><span class="p">,</span> <span class="n">train</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">download</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
    189. 

  189.                                         <span class="n">transform</span><span class="o">=</span><span class="n">transform_train</span><span class="p">)</span>
    190. 

  190. 

  191.        <span class="bp">self</span><span class="o">.</span><span class="n">valset</span> <span class="o">=</span> <span class="n">datasets</span><span class="o">.</span><span class="n">CIFAR10</span><span class="p">(</span><span class="n">root</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">dataset_params</span><span class="o">.</span><span class="n">dataset_dir</span><span class="p">,</span> <span class="n">train</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">download</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
    192. 

  192.                                        <span class="n">transform</span><span class="o">=</span><span class="n">transform_val</span><span class="p">)</span>
    193. 

  193. 

  194. </pre></div>
    195. 

  195. </div>
    196. 

  196. <p>Required parameters can be passed using the <code class="docutils literal notranslate"><span class="pre">python</span> <span class="pre">dataset_params</span></code> argument. When implementing a dataset interface, the<code class="docutils literal notranslate"><span class="pre">trainset</span></code> and <code class="docutils literal notranslate"><span class="pre">valset</span></code> attributes are required and must be initiated with a <em>torch.utils.data.Dataset</em> type. These fields will cause the <em>SgModule</em> instance to use them accordingly, such as during training, testing, and so on.</p>
    197. 

  197. </section>
    198. 

  198. <section id="integrating-your-training-code-complete-walkthrough-model">
    199. 

  199. <h2>Integrating Your Training Code - Complete Walkthrough: Model<a class="headerlink" href="#integrating-your-training-code-complete-walkthrough-model" title="Permalink to this headline"></a></h2>
    200. 

  200. <p>This is rather straightforward- the only requirement is that the model must be of torch.nn.Module type. In our case, a simple Lenet implementation (taken from https://github.com/icpm/pytorch-cifar10/blob/master/models/LeNet.py).</p>
    201. 

  201. <table>
    202. 

  202.   <tr>
    203. 

  203.   </tr>
    204. 

  204. </table>
    205. 

  205. <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">torch.nn</span> <span class="k">as</span> <span class="nn">nn</span>
    206. 

  206. <span class="kn">import</span> <span class="nn">torch.nn.functional</span> <span class="k">as</span> <span class="nn">func</span>
    207. 

  207. 

  208. 

  209. <span class="k">class</span> <span class="nc">LeNet</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">Module</span><span class="p">):</span>
    210. 

  210.    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
    211. 

  211.        <span class="nb">super</span><span class="p">(</span><span class="n">LeNet</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
    212. 

  212.        <span class="bp">self</span><span class="o">.</span><span class="n">conv1</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Conv2d</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="n">kernel_size</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>
    213. 

  213.        <span class="bp">self</span><span class="o">.</span><span class="n">conv2</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Conv2d</span><span class="p">(</span><span class="mi">6</span><span class="p">,</span> <span class="mi">16</span><span class="p">,</span> <span class="n">kernel_size</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>
    214. 

  214.        <span class="bp">self</span><span class="o">.</span><span class="n">fc1</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="mi">16</span><span class="o">*</span><span class="mi">5</span><span class="o">*</span><span class="mi">5</span><span class="p">,</span> <span class="mi">120</span><span class="p">)</span>
    215. 

  215.        <span class="bp">self</span><span class="o">.</span><span class="n">fc2</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="mi">120</span><span class="p">,</span> <span class="mi">84</span><span class="p">)</span>
    216. 

  216.        <span class="bp">self</span><span class="o">.</span><span class="n">fc3</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">Linear</span><span class="p">(</span><span class="mi">84</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span>
    217. 

  217. 

  218.    <span class="k">def</span> <span class="nf">forward</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span>
    219. 

  219.        <span class="n">x</span> <span class="o">=</span> <span class="n">func</span><span class="o">.</span><span class="n">relu</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">conv1</span><span class="p">(</span><span class="n">x</span><span class="p">))</span>
    220. 

  220.        <span class="n">x</span> <span class="o">=</span> <span class="n">func</span><span class="o">.</span><span class="n">max_pool2d</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
    221. 

  221.        <span class="n">x</span> <span class="o">=</span> <span class="n">func</span><span class="o">.</span><span class="n">relu</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">conv2</span><span class="p">(</span><span class="n">x</span><span class="p">))</span>
    222. 

  222.        <span class="n">x</span> <span class="o">=</span> <span class="n">func</span><span class="o">.</span><span class="n">max_pool2d</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
    223. 

  223.        <span class="n">x</span> <span class="o">=</span> <span class="n">x</span><span class="o">.</span><span class="n">view</span><span class="p">(</span><span class="n">x</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="mi">0</span><span class="p">),</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span>
    224. 

  224.        <span class="n">x</span> <span class="o">=</span> <span class="n">func</span><span class="o">.</span><span class="n">relu</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">fc1</span><span class="p">(</span><span class="n">x</span><span class="p">))</span>
    225. 

  225.        <span class="n">x</span> <span class="o">=</span> <span class="n">func</span><span class="o">.</span><span class="n">relu</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">fc2</span><span class="p">(</span><span class="n">x</span><span class="p">))</span>
    226. 

  226.        <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">fc3</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
    227. 

  227.        <span class="k">return</span> <span class="n">x</span>
    228. 

  228. 

  229. </pre></div>
    230. 

  230. </div>
    231. 

  231. </section>
    232. 

  232. <section id="integrating-your-training-code-complete-walkthrough-loss-function">
    233. 

  233. <h2>Integrating Your Training Code - Complete Walkthrough: Loss Function<a class="headerlink" href="#integrating-your-training-code-complete-walkthrough-loss-function" title="Permalink to this headline"></a></h2>
    234. 

  234. <p>The loss function class must be of _torch.nn.module.<em>LOSS</em> type. For example, our _LabelSmoothingCrossEntropyLoss _implementation.</p>
    235. 

  235. <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">torch.nn</span> <span class="k">as</span> <span class="nn">nn</span>
    236. 

  236. <span class="kn">from</span> <span class="nn">super_gradients.training.losses.label_smoothing_cross_entropy_loss</span> <span class="kn">import</span> <span class="n">cross_entropy</span>
    237. 

  237. 

  238. <span class="k">class</span> <span class="nc">LabelSmoothingCrossEntropyLoss</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">CrossEntropyLoss</span><span class="p">):</span>
    239. 

  239.    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">weight</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">ignore_index</span><span class="o">=-</span><span class="mi">100</span><span class="p">,</span> <span class="n">reduction</span><span class="o">=</span><span class="s1">&#39;mean&#39;</span><span class="p">,</span> <span class="n">smooth_eps</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">smooth_dist</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
    240. 

  240.                 <span class="n">from_logits</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
    241. 

  241.        <span class="nb">super</span><span class="p">(</span><span class="n">LabelSmoothingCrossEntropyLoss</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">weight</span><span class="o">=</span><span class="n">weight</span><span class="p">,</span>
    242. 

  242.                                                             <span class="n">ignore_index</span><span class="o">=</span><span class="n">ignore_index</span><span class="p">,</span> <span class="n">reduction</span><span class="o">=</span><span class="n">reduction</span><span class="p">)</span>
    243. 

  243.        <span class="bp">self</span><span class="o">.</span><span class="n">smooth_eps</span> <span class="o">=</span> <span class="n">smooth_eps</span>
    244. 

  244.        <span class="bp">self</span><span class="o">.</span><span class="n">smooth_dist</span> <span class="o">=</span> <span class="n">smooth_dist</span>
    245. 

  245.        <span class="bp">self</span><span class="o">.</span><span class="n">from_logits</span> <span class="o">=</span> <span class="n">from_logits</span>
    246. 

  246. 

  247.    <span class="k">def</span> <span class="nf">forward</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">input</span><span class="p">,</span> <span class="n">target</span><span class="p">,</span> <span class="n">smooth_dist</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
    248. 

  248.        <span class="k">if</span> <span class="n">smooth_dist</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
    249. 

  249.            <span class="n">smooth_dist</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">smooth_dist</span>
    250. 

  250.        <span class="n">loss</span> <span class="o">=</span> <span class="n">cross_entropy</span><span class="p">(</span><span class="nb">input</span><span class="p">,</span> <span class="n">target</span><span class="p">,</span> <span class="n">weight</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">weight</span><span class="p">,</span> <span class="n">ignore_index</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">ignore_index</span><span class="p">,</span>
    251. 

  251.                             <span class="n">reduction</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">reduction</span><span class="p">,</span> <span class="n">smooth_eps</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">smooth_eps</span><span class="p">,</span>
    252. 

  252.                             <span class="n">smooth_dist</span><span class="o">=</span><span class="n">smooth_dist</span><span class="p">,</span> <span class="n">from_logits</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">from_logits</span><span class="p">)</span>
    253. 

  253. 

  254.        <span class="k">return</span> <span class="n">loss</span>
    255. 

  255. 

  256. </pre></div>
    257. 

  257. </div>
    258. 

  258. <p><strong>Important –</strong> <em>forward(…)</em> may return a (loss, loss_items) tuple instead of just a single item (i.e loss), where –</p>
    259. 

  259. <p><em>loss</em> is the tensor used for backprop, meaning what your original loss function returns.</p>
    260. 

  260. <p><em>loss_items</em> must be a tensor of shape (n_items) that is composed of values that are computed during the forward pass, so that it can be logged over the entire epoch.</p>
    261. 

  261. <p>For example, the loss itself should always be logged. Another example is a scenario where the computed loss is the sum of a few components. These entries should be logged in loss_items.</p>
    262. 

  262. <p>During training, set the <em><span style="text-decoration:underline;">loss_logging_items_names</span></em> parameter in _<span style="text-decoration:underline;">training_params</span> _to be a list of strings of length <em>n_items</em>, whose ith element is the name of the ith entry in loss_items. In this way, each item will be logged, rendered and monitored in TensorBoard, thus saving model checkpoints accordingly.</p>
    263. 

  263. <p>Because running logs save the loss_items in some internal state. It is therefore recommended that loss_items be detached from their computational graph for memory efficiency.</p>
    264. 

  264. </section>
    265. 

  265. <section id="integrating-your-training-code-complete-walkthrough-metrics">
    266. 

  266. <h2>Integrating Your Training Code - Complete Walkthrough: Metrics<a class="headerlink" href="#integrating-your-training-code-complete-walkthrough-metrics" title="Permalink to this headline"></a></h2>
    267. 

  267. <p>The metrics objects to be logged during training must be of torchmetrics.Metric type. For more information on how to use torchmetric.Metric objects and implement your own metrics. see https://torchmetrics.readthedocs.io/en/latest/pages/overview.html.</p>
    268. 

  268. <p>During training, the metric’s update is called with the model’s raw outputs and raw targets. Therefore, any processing of the two must be taken into account and applied in the <em>update</em>.</p>
    269. 

  269. <p>Training works out of the box with any of the module torchmetrics (full list in <a class="reference external" href="https://torchmetrics.readthedocs.io/en/latest/references/modules.html">https://torchmetrics.readthedocs.io/en/latest/references/modules.html</a>). Additional metrics implementations such as mean average precision for object detection can be found at <em>super_gradients.training.metrics</em>)</p>
    270. 

  270. <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">torchmetrics</span>
    271. 

  271. <span class="kn">import</span> <span class="nn">torch</span>
    272. 

  272. 

  273. 

  274. <span class="k">class</span> <span class="nc">Accuracy</span><span class="p">(</span><span class="n">torchmetrics</span><span class="o">.</span><span class="n">Accuracy</span><span class="p">):</span>
    275. 

  275.    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">dist_sync_on_step</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
    276. 

  276.        <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">dist_sync_on_step</span><span class="o">=</span><span class="n">dist_sync_on_step</span><span class="p">,</span> <span class="n">top_k</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
    277. 

  277. 

  278.    <span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">preds</span><span class="p">:</span> <span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">,</span> <span class="n">target</span><span class="p">:</span> <span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">):</span>
    279. 

  279.        <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">preds</span><span class="o">=</span><span class="n">preds</span><span class="o">.</span><span class="n">softmax</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span> <span class="n">target</span><span class="o">=</span><span class="n">target</span><span class="p">)</span>
    280. 

  280. 

  281. 

  282. <span class="k">class</span> <span class="nc">Top5</span><span class="p">(</span><span class="n">torchmetrics</span><span class="o">.</span><span class="n">Accuracy</span><span class="p">):</span>
    283. 

  283.    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">dist_sync_on_step</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
    284. 

  284.        <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">dist_sync_on_step</span><span class="o">=</span><span class="n">dist_sync_on_step</span><span class="p">,</span> <span class="n">top_k</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>
    285. 

  285. 

  286.    <span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">preds</span><span class="p">:</span> <span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">,</span> <span class="n">target</span><span class="p">:</span> <span class="n">torch</span><span class="o">.</span><span class="n">Tensor</span><span class="p">):</span>
    287. 

  287.        <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">preds</span><span class="o">=</span><span class="n">preds</span><span class="o">.</span><span class="n">softmax</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span> <span class="n">target</span><span class="o">=</span><span class="n">target</span><span class="p">)</span>
    288. 

  288. </pre></div>
    289. 

  289. </div>
    290. 

  290. </section>
    291. 

  291. <section id="integrating-your-training-code-complete-walkthrough-training-script">
    292. 

  292. <h2>Integrating Your Training Code- Complete Walkthrough: Training script<a class="headerlink" href="#integrating-your-training-code-complete-walkthrough-training-script" title="Permalink to this headline"></a></h2>
    293. 

  293. <p>We instantiate an SgModel and a UserDatasetInterface, then call connect_dataset_interface which will initialize the dataloaders and pass additional dataset parameters to the SgModel instance.</p>
    294. 

  294. <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">super_gradients.training</span> <span class="kn">import</span> <span class="n">SgModel</span>
    295. 

  295. 

  296. <span class="n">sg_model</span> <span class="o">=</span> <span class="n">SgModel</span><span class="p">(</span><span class="n">experiment_name</span><span class="o">=</span><span class="s1">&#39;LeNet_cifar10_example&#39;</span><span class="p">)</span>
    297. 

  297. <span class="n">dataset_params</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&quot;batch_size&quot;</span><span class="p">:</span> <span class="mi">256</span><span class="p">}</span>
    298. 

  298. <span class="n">dataset</span> <span class="o">=</span> <span class="n">UserDataset</span><span class="p">(</span><span class="n">dataset_params</span><span class="p">)</span>
    299. 

  299. <span class="n">sg_model</span><span class="o">.</span><span class="n">connect_dataset_interface</span><span class="p">(</span><span class="n">dataset</span><span class="p">)</span>
    300. 

  300. 

  301. </pre></div>
    302. 

  302. </div>
    303. 

  303. <p><strong>Now, we pass a LeNet instance we defined above to the SgModel:</strong></p>
    304. 

  304. <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">network</span> <span class="o">=</span> <span class="n">LeNet</span><span class="p">()</span>
    305. 

  305. <span class="n">sg_model</span><span class="o">.</span><span class="n">build_model</span><span class="p">(</span><span class="n">network</span><span class="p">)</span>
    306. 

  306. </pre></div>
    307. 

  307. </div>
    308. 

  308. <p><strong>Next, we define metrics in order to evaluate our model.</strong></p>
    309. 

  309. <div class="highlight-default notranslate"><div class="highlight"><pre><span></span> <span class="kn">from</span> <span class="nn">super_gradients.training.metrics</span> <span class="kn">import</span> <span class="n">Accuracy</span><span class="p">,</span> <span class="n">Top5</span>
    310. 

  310. 

  311. <span class="n">train_metrics_list</span> <span class="o">=</span> <span class="p">[</span><span class="n">Accuracy</span><span class="p">(),</span> <span class="n">Top5</span><span class="p">()]</span>
    312. 

  312. <span class="n">valid_metrics_list</span> <span class="o">=</span> <span class="p">[</span><span class="n">Accuracy</span><span class="p">(),</span> <span class="n">Top5</span><span class="p">()]</span>
    313. 

  313. 

  314. </pre></div>
    315. 

  315. </div>
    316. 

  316. <p>Initializing the loss, and specifying training parameters</p>
    317. 

  317. <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">train_params</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&quot;max_epochs&quot;</span><span class="p">:</span> <span class="mi">250</span><span class="p">,</span>
    318. 

  318.                <span class="s2">&quot;lr_updates&quot;</span><span class="p">:</span> <span class="p">[</span><span class="mi">100</span><span class="p">,</span> <span class="mi">150</span><span class="p">,</span> <span class="mi">200</span><span class="p">],</span>
    319. 

  319.                <span class="s2">&quot;lr_decay_factor&quot;</span><span class="p">:</span> <span class="mf">0.1</span><span class="p">,</span>
    320. 

  320.                <span class="s2">&quot;lr_mode&quot;</span><span class="p">:</span> <span class="s2">&quot;step&quot;</span><span class="p">,</span>
    321. 

  321.                <span class="s2">&quot;lr_warmup_epochs&quot;</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
    322. 

  322.                <span class="s2">&quot;initial_lr&quot;</span><span class="p">:</span> <span class="mf">0.1</span><span class="p">,</span>
    323. 

  323.                <span class="s2">&quot;loss&quot;</span><span class="p">:</span> <span class="n">LabelSmoothingCrossEntropyLoss</span><span class="p">(),</span>
    324. 

  324.                <span class="s2">&quot;criterion_params&quot;</span><span class="p">:</span> <span class="p">{},</span>
    325. 

  325.                <span class="s2">&quot;optimizer&quot;</span><span class="p">:</span> <span class="s2">&quot;SGD&quot;</span><span class="p">,</span>
    326. 

  326.                <span class="s2">&quot;optimizer_params&quot;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&quot;weight_decay&quot;</span><span class="p">:</span> <span class="mf">1e-4</span><span class="p">,</span> <span class="s2">&quot;momentum&quot;</span><span class="p">:</span><span class="mf">0.9</span><span class="p">},</span>
    327. 

  327.                <span class="s2">&quot;launch_tensorboard&quot;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
    328. 

  328.                <span class="s2">&quot;train_metrics_list&quot;</span><span class="p">:</span> <span class="n">train_metrics_list</span><span class="p">,</span>
    329. 

  329.                <span class="s2">&quot;valid_metrics_list&quot;</span><span class="p">:</span> <span class="n">valid_metrics_list</span><span class="p">,</span>
    330. 

  330.                <span class="s2">&quot;loss_logging_items_names&quot;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&quot;Loss&quot;</span><span class="p">],</span>
    331. 

  331.                <span class="s2">&quot;metric_to_watch&quot;</span><span class="p">:</span> <span class="s2">&quot;Accuracy&quot;</span><span class="p">,</span>
    332. 

  332.                <span class="s2">&quot;greater_metric_to_watch_is_better&quot;</span><span class="p">:</span> <span class="kc">True</span><span class="p">}</span>
    333. 

  333. 

  334. <span class="n">sg_model</span><span class="o">.</span><span class="n">train</span><span class="p">(</span><span class="n">train_params</span><span class="p">)</span>
    335. 

  335. 

  336. </pre></div>
    337. 

  337. </div>
    338. 

  338. <section id="training-parameter-notes">
    339. 

  339. <h3>Training Parameter Notes:<a class="headerlink" href="#training-parameter-notes" title="Permalink to this headline"></a></h3>
    340. 

  340. <ul class="simple">
    341. 

  341. <li><p>_<span style="text-decoration:underline;">loss_logging_items_names</span> _parameter – Refers to the single item returned in <em>loss_items</em> in our loss function described above.</p></li>
    342. 

  342. <li><p><em><span style="text-decoration:underline;">metric_to_watch</span></em> – Is the model’s metric that determines the checkpoint to be saved. In our example, this parameter is set to <em>Accuracy</em>, and can be set to any of the following:</p></li>
    343. 

  343. <li><p>A metric name (str) of one of the metric objects from the _valid_metrics_lis_t.</p></li>
    344. 

  344. <li><p>A <em>metric_name</em> that represents a metric that appears in <em>valid_metrics_list</em> and has an attribute <em>component_names</em>. <em>component_names</em> is a list that refers to the names of each entry in the output metric (torch tensor of size n).</p></li>
    345. 

  345. <li><p>One of the <em>loss_logging_items_names</em>, such as one that corresponds to an item returned during the loss function’s forward pass as discussed earlier.</p></li>
    346. 

  346. <li><p><em><span style="text-decoration:underline;">greater_metric_to_watch_is_better flag </span></em>– Determines when to save a model’s checkpoint according to the value of the <code class="docutils literal notranslate"><span class="pre">metric_to_watch</span></code>.</p></li>
    347. 

  347. </ul>
    348. 

  348. </section>
    349. 

  349. </section>
    350. 

  350. </section>
    351. 

  351. <section id="training-parameters">
    352. 

  352. <h1>Training Parameters<a class="headerlink" href="#training-parameters" title="Permalink to this headline"></a></h1>
    353. 

  353. <p>The following is a description of all the parameters passed in _training_params _when _<span style="text-decoration:underline;">train() </span>_is called.</p>
    354. 

  354. <p><code class="docutils literal notranslate"><span class="pre">max_epochs</span></code>: int</p>
    355. 

  355. <p>Number of epochs to run during training.</p>
    356. 

  356. <p><code class="docutils literal notranslate"><span class="pre">lr_updates</span></code>: list(int)</p>
    357. 

  357. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>List of fixed epoch numbers to perform learning rate updates when `lr_mode=&#39;step&#39;`.
    358. 

  358. </pre></div>
    359. 

  359. </div>
    360. 

  360. <p><code class="docutils literal notranslate"><span class="pre">lr_decay_factor</span></code>: float</p>
    361. 

  361. <p>Decay factor to apply to the learning rate at each update when <em>lr_mode=’step’</em>.</p>
    362. 

  362. <p><code class="docutils literal notranslate"><span class="pre">lr_mode</span></code>: str</p>
    363. 

  363. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>Learning rate scheduling policy, one of [&#39;step&#39;,&#39;poly&#39;,&#39;cosine&#39;,&#39;function&#39;].
    364. 

  364. </pre></div>
    365. 

  365. </div>
    366. 

  366. <ul class="simple">
    367. 

  367. <li><p>‘step’ refers to constant updates of epoch numbers passed through <code class="docutils literal notranslate"><span class="pre">lr_updates</span></code>.</p></li>
    368. 

  368. <li><p>‘cosine’ refers to Cosine Annealing policy as described in https://arxiv.org/abs/1608.03983.</p></li>
    369. 

  369. <li><p>‘poly’ refers to polynomial decrease, such as in each epoch iteration <code class="docutils literal notranslate"><span class="pre">self.lr</span> <span class="pre">=</span> <span class="pre">self.initial_lr</span> <span class="pre">*</span> <span class="pre">pow((1.0</span> <span class="pre">-</span> <span class="pre">(current_iter</span> <span class="pre">/</span> <span class="pre">max_iter)),</span> <span class="pre">0.9)</span></code></p></li>
    370. 

  370. <li><p>‘function’ refers to a user defined learning rate scheduling function, that is passed through <code class="docutils literal notranslate"><span class="pre">lr_schedule_function</span></code>.</p></li>
    371. 

  371. </ul>
    372. 

  372. <p><code class="docutils literal notranslate"><span class="pre">lr_schedule_function</span></code>: Union[callable,None]</p>
    373. 

  373. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>Learning rate scheduling function to be used when `lr_mode` is &#39;function&#39;.
    374. 

  374. </pre></div>
    375. 

  375. </div>
    376. 

  376. <p><code class="docutils literal notranslate"><span class="pre">lr_warmup_epochs</span></code>: int (default=0)</p>
    377. 

  377. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>Number of epochs for learning rate warm up. For more information, you may refer to https://arxiv.org/pdf/1706.02677.pdf (Section 2.2).
    378. 

  378. </pre></div>
    379. 

  379. </div>
    380. 

  380. <p><code class="docutils literal notranslate"><span class="pre">cosine_final_lr_ratio</span></code>: float (default=0.01)</p>
    381. 

  381. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>Final learning rate ratio (only relevant when `lr_mode`=&#39;cosine&#39;). The cosine starts from initial_lr and reaches initial_lr * cosine_final_lr_ratio in the last epoch.
    382. 

  382. </pre></div>
    383. 

  383. </div>
    384. 

  384. <p><code class="docutils literal notranslate"><span class="pre">inital_lr</span></code>: float</p>
    385. 

  385. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>Initial learning rate.
    386. 

  386. </pre></div>
    387. 

  387. </div>
    388. 

  388. <p><code class="docutils literal notranslate"><span class="pre">loss</span></code>: Union[nn.module, str]</p>
    389. 

  389. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>Loss function to be used for training.
    390. 

  390. 

  391. 

  392. One of super_gradients&#39;s built in options:
    393. 

  393. 

  394. 

  395.               &quot;cross_entropy&quot;: LabelSmoothingCrossEntropyLoss,
    396. 

  396. 

  397. 

  398.               &quot;mse&quot;: MSELoss,
    399. 

  399. 

  400. 

  401.               &quot;r_squared_loss&quot;: RSquaredLoss,
    402. 

  402. 

  403. 

  404.               &quot;detection_loss&quot;: YoLoV3DetectionLoss,
    405. 

  405. 

  406. 

  407.               &quot;shelfnet_ohem_loss&quot;: ShelfNetOHEMLoss,
    408. 

  408. 

  409. 

  410.               &quot;shelfnet_se_loss&quot;: ShelfNetSemanticEncodingLoss,
    411. 

  411. 

  412. 

  413.               &quot;yolo_v5_loss&quot;: YoLoV5DetectionLoss,
    414. 

  414. 

  415. 

  416.               &quot;ssd_loss&quot;: SSDLoss,
    417. 

  417. 

  418. 

  419.         or user defined nn.module loss function.
    420. 

  420. 

  421. 

  422. **Important –** _forward(...)_ should return a (loss, loss_items) tuple, where –
    423. 

  423. </pre></div>
    424. 

  424. </div>
    425. 

  425. <ul>
    426. 

  426. <li><p><em>loss</em> is the tensor used for backprop, meaning what your original loss function returns</p></li>
    427. 

  427. <li><p><em>loss_items</em> must be a tensor of shape (n_items) of values computed during the forward pass, so that they can be logged over the entire epoch.</p>
    428. 

  428. <p>For example, the loss itself should always be logged. Another example is a scenario where the computed loss is the sum of a few components. These entries should be returned in loss_items.</p>
    429. 

  429. <p>During training, set the <em>loss_logging_items_names</em> parameter in _training_params _to be a list of strings of length <em>n_items</em>, whose ith element is the name of the ith entry in loss_items. In this way, each item will be logged, rendered on TensorBoard and monitored, thus saving model checkpoints accordingly.</p>
    430. 

  430. <p>Running logs saves the loss_items in some internal state. It is therefore recommended that loss_items be detached from their computational graph for memory efficiency.</p>
    431. 

  431. </li>
    432. 

  432. </ul>
    433. 

  433. <p><code class="docutils literal notranslate"><span class="pre">optimizer</span></code>: str</p>
    434. 

  434. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>Optimization algorithm. One of [&#39;Adam&#39;,&#39;SGD&#39;,&#39;RMSProp&#39;] corresponding to the torch.optim optimzer implementations.
    435. 

  435. </pre></div>
    436. 

  436. </div>
    437. 

  437. <p><code class="docutils literal notranslate"><span class="pre">criterion_params</span></code>: dict</p>
    438. 

  438. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>Loss function parameters.
    439. 

  439. </pre></div>
    440. 

  440. </div>
    441. 

  441. <p><code class="docutils literal notranslate"><span class="pre">optimizer_params</span></code>: dict</p>
    442. 

  442. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>Optimizer parameters. You may refer to https://pytorch.org/docs/stable/optim.html for the full list of the parameters for each optimizer.
    443. 

  443. </pre></div>
    444. 

  444. </div>
    445. 

  445. <p><code class="docutils literal notranslate"><span class="pre">train_metrics_list</span></code>: list(torchmetrics.Metric)</p>
    446. 

  446. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>Metrics to log during training. You may refer to [https://torchmetrics.rtfd.io/en/latest/](https://torchmetrics.rtfd.io/en/latest/), for more information about TorchMetrics.
    447. 

  447. </pre></div>
    448. 

  448. </div>
    449. 

  449. <p><code class="docutils literal notranslate"><span class="pre">valid_metrics_list</span></code>: list(torchmetrics.Metric)</p>
    450. 

  450. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>Metrics to log during validation/testing. You may refer to [https://torchmetrics.rtfd.io/en/latest/](https://torchmetrics.rtfd.io/en/latest/), for more information about TorchMetrics.
    451. 

  451. </pre></div>
    452. 

  452. </div>
    453. 

  453. <p><code class="docutils literal notranslate"><span class="pre">loss_logging_items_names</span></code>: list(str)</p>
    454. 

  454. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>The list of names/titles for the outputs returned from the loss function’s forward pass. These names are used to log their values.
    455. 

  455. 

  456. 

  457. **Note – **The loss function should return the tuple (loss, loss_items).
    458. 

  458. </pre></div>
    459. 

  459. </div>
    460. 

  460. <p><code class="docutils literal notranslate"><span class="pre">metric_to_watch</span></code>: str (default=”Accuracy”)</p>
    461. 

  461. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>Specifies the metric according to which the model checkpoint is saved. It can be set to any of the following:
    462. 

  462. </pre></div>
    463. 

  463. </div>
    464. 

  464. <ul>
    465. 

  465. <li><p>A metric name (str) of one of the metric objects from the valid_metrics_list</p></li>
    466. 

  466. <li><p>A “metric_name” to be used if any metric in the valid_metrics_list has an attribute component_names, which is a list referring to the names of each entry in the output metric (torch tensor of size n).</p></li>
    467. 

  467. <li><p>One of the “loss_logging_items_names” <code class="docutils literal notranslate"><span class="pre">that</span></code> corresponds to an item to be returned during the loss function’s forward pass.</p>
    468. 

  468. <p>At the end of each epoch, if a new best _metric_to_watch _value is achieved, the model’s checkpoint is saved in YOUR_PYTHON_PATH/checkpoints/ckpt_best.pth.</p>
    469. 

  469. </li>
    470. 

  470. </ul>
    471. 

  471. <p><code class="docutils literal notranslate"><span class="pre">greater_metric_to_watch_is_better</span></code>: bool</p>
    472. 

  472. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>Determines when to save a model&#39;s checkpoint according to the value of the` metric_to_watch:`
    473. 

  473. </pre></div>
    474. 

  474. </div>
    475. 

  475. <ul class="simple">
    476. 

  476. <li><p>_True: _A model’s checkpoint is saved when the model achieves the highest metric_to_watch.</p></li>
    477. 

  477. <li><p><em>False:</em> A model’s checkpoint is saved when the model achieves the lowest metric_to_watch.</p></li>
    478. 

  478. </ul>
    479. 

  479. <p><code class="docutils literal notranslate"><span class="pre">ema</span></code>: bool (default=False)</p>
    480. 

  480. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>Specifies whether to use Model Exponential Moving Average. You may refer to https://github.com/rwightman/pytorch-image-models ema implementation), for more information.
    481. 

  481. </pre></div>
    482. 

  482. </div>
    483. 

  483. <p><code class="docutils literal notranslate"><span class="pre">batch_accumulate</span></code>: int (default=1)</p>
    484. 

  484. <p>Number of batches to accumulate before every backward pass.</p>
    485. 

  485. <p><code class="docutils literal notranslate"><span class="pre">ema_params</span></code>: dict</p>
    486. 

  486. <p>Parameters for the ema model.</p>
    487. 

  487. <p><code class="docutils literal notranslate"><span class="pre">zero_weight_decay_on_bias_and_bn</span></code>: bool (default=False)</p>
    488. 

  488. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>Specifies whether to apply weight decay on batch normalization parameters or not.
    489. 

  489. </pre></div>
    490. 

  490. </div>
    491. 

  491. <p><code class="docutils literal notranslate"><span class="pre">load_opt_params</span></code>: bool (default=True)</p>
    492. 

  492. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>Specifies whether to load the optimizers parameters (as well) when loading a model&#39;s checkpoint.
    493. 

  493. </pre></div>
    494. 

  494. </div>
    495. 

  495. <p><code class="docutils literal notranslate"><span class="pre">run_validation_freq</span></code>: int (default=1)</p>
    496. 

  496. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>The frequency at which validation is performed during training. This means that  the validation is run every `run_validation_freq` epochs.
    497. 

  497. </pre></div>
    498. 

  498. </div>
    499. 

  499. <p><code class="docutils literal notranslate"><span class="pre">save_model</span></code>: bool (default=True)</p>
    500. 

  500. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span> Specifies whether to save the model’s checkpoints.
    501. 

  501. </pre></div>
    502. 

  502. </div>
    503. 

  503. <p><code class="docutils literal notranslate"><span class="pre">launch_tensorboard</span></code>: bool (default=False)</p>
    504. 

  504. <p>Specifies whether to launch a TensorBoard process.</p>
    505. 

  505. <p><code class="docutils literal notranslate"><span class="pre">tb_files_user_prompt</span></code>: bool</p>
    506. 

  506. <p>Displays the TensorBoard deletion user prompt.</p>
    507. 

  507. <p><code class="docutils literal notranslate"><span class="pre">silent_mode</span></code>: bool</p>
    508. 

  508. <p>Deactivates the printouts.</p>
    509. 

  509. <p><code class="docutils literal notranslate"><span class="pre">mixed_precision</span></code>: bool</p>
    510. 

  510. <p>Specifies whether to use mixed precision or not.</p>
    511. 

  511. <p><code class="docutils literal notranslate"><span class="pre">tensorboard_port</span></code>: int, None (default=None)</p>
    512. 

  512. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>Specific port number for the TensorBoard to use when launched (when set to None, some free port number will be used).
    513. 

  513. </pre></div>
    514. 

  514. </div>
    515. 

  515. <p><code class="docutils literal notranslate"><span class="pre">save_ckpt_epoch_list</span></code>: list(int) (default=[])</p>
    516. 

  516. <p>Specifies the list of fixed epoch indices in which to save checkpoints.</p>
    517. 

  517. <p><code class="docutils literal notranslate"><span class="pre">average_best_models</span></code>: bool (default=False)</p>
    518. 

  518. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>If True, a snapshot dictionary file and the average model will be saved / updated at every epoch and only evaluated after the training has completed. The snapshot file will only be deleted upon completing the training. The snapshot dict will be managed on the CPU.
    519. 

  519. </pre></div>
    520. 

  520. </div>
    521. 

  521. <p><code class="docutils literal notranslate"><span class="pre">save_tensorboard_to_s3</span></code>: bool (default=False)</p>
    522. 

  522. <p>If True,  saves the TensorBoard in S3.</p>
    523. 

  523. <p><code class="docutils literal notranslate"><span class="pre">precise_bn</span></code>: bool (default=False)</p>
    524. 

  524. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>   Whether to use precise_bn calculation during the training.
    525. 

  525. </pre></div>
    526. 

  526. </div>
    527. 

  527. <p><code class="docutils literal notranslate"><span class="pre">precise_bn_batch_size</span></code>: int (default=None)</p>
    528. 

  528. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>The effective batch size we want to calculate the batchnorm on. For example, if  we are training a model on 8 gpus, with a batch of 128 on each gpu, a good rule of thumb would be to give it 8192 (ie: effective_batch_size * num_gpus = batch_per_gpu * m_gpus * num_gpus). If precise_bn_batch_size is not provided in the training_params, the latter heuristic will be taken.
    529. 

  529. </pre></div>
    530. 

  530. </div>
    531. 

  531. <p><code class="docutils literal notranslate"><span class="pre">seed</span></code> : int (default=42)</p>
    532. 

  532. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>Random seed to be set for torch, numpy, and random. When using DDP each process will have it&#39;s seed set to seed + rank.
    533. 

  533. </pre></div>
    534. 

  534. </div>
    535. 

  535. <p><code class="docutils literal notranslate"><span class="pre">log_installed_packages</span></code>: bool (default=False)</p>
    536. 

  536. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>When set, the list of all installed packages (and their versions) will be written to the tensorboard and logfile (useful when trying to reproduce results).
    537. 

  537. </pre></div>
    538. 

  538. </div>
    539. 

  539. <p><code class="docutils literal notranslate"><span class="pre">dataset_statistics</span></code>:: bool (default=False)</p>
    540. 

  540. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>Enable a statistic analysis of the dataset. If set to True the dataset will be analyzed and a report   will be added to the tensorboard along with some sample images from the dataset. Currently only detection datasets are supported for analysis.
    541. 

  541. </pre></div>
    542. 

  542. </div>
    543. 

  543. <p><code class="docutils literal notranslate"><span class="pre">save_full_train_log</span></code> : bool (default=False)</p>
    544. 

  544. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>When set, a full log (of all super_gradients modules, including uncaught exceptions from any other module) of the training will be saved in the checkpoint directory under full_train_log.log
    545. 

  545. </pre></div>
    546. 

  546. </div>
    547. 

  547. </section>
    548. 

  548. <section id="logs-and-checkpoints">
    549. 

  549. <h1>Logs and Checkpoints<a class="headerlink" href="#logs-and-checkpoints" title="Permalink to this headline"></a></h1>
    550. 

  550. <p>The model’s weights, logs and tensorboards are saved in _”YOUR_PYTHONPATH”/ checkpoints/”YOUR_EXPERIMENT_NAME” _. (In our  walkthrough example, _”YOUR_EXPERIMENT_NAME” _ is <em>user_model_training)</em>.</p>
    551. 

  551. <ul class="simple">
    552. 

  552. <li></li>
    553. 

  553. </ul>
    554. 

  554. <p><strong>To watch training progress –</strong></p>
    555. 

  555. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>**1st option:**
    556. 

  556. </pre></div>
    557. 

  557. </div>
    558. 

  558. <ol class="arabic">
    559. 

  559. <li><p>Open a terminal.</p></li>
    560. 

  560. <li><p>Navigate to _”YOUR_LOCAL_PATH_TO_super_gradients_PACKAGE”/ _and run ``tensorboard –logdir checkpoints –bind_all`.</p>
    561. 

  561. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>     The message `TensorBoard 2.4.1 at http://localhost:XXXX/` appears.
    562. 

  562. </pre></div>
    563. 

  563. </div>
    564. 

  564. </li>
    565. 

  565. <li><p>Follow the link in this message to see the progress of the training.</p>
    566. 

  566. <p><strong>2nd option:</strong></p>
    567. 

  567. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span> Set the “launch_tensorboard_process” flag in your training_params passed to SgModel.train(...), and follow instructions displayed in the shell.
    568. 

  568. </pre></div>
    569. 

  569. </div>
    570. 

  570. </li>
    571. 

  571. </ol>
    572. 

  572. <ul class="simple">
    573. 

  573. <li></li>
    574. 

  574. </ul>
    575. 

  575. <p><strong>To resume training –</strong>
    576. 

  576. When building the network- call SgModel.build_model(…load_checkpoint=True). Doing so, will load the network’s weights, as well as any relevant information for resuming training (monitored metric values, optimizer states, etc) with the latest checkpoint. For more advanced usage see SgModel.build_model docs in code.</p>
    577. 

  577. <ul class="simple">
    578. 

  578. <li></li>
    579. 

  579. </ul>
    580. 

  580. <p><strong>Checkpoint structure – state_dict (see <a class="reference external" href="https://pytorch.org/tutorials/beginner/saving_loading_models.html">https://pytorch.org/tutorials/beginner/saving_loading_models.html</a> for more information regarding state_dicts) with the following keys:</strong>
    581. 

  581. <strong>-”net”-  The network’s state_dict.</strong></p>
    582. 

  582. <p><strong>-”acc”-  The value of <code class="docutils literal notranslate"><span class="pre">metric_to_watch</span></code> from training.</strong></p>
    583. 

  583. <p><strong>-”epoch”- Last epoch performed before saving this checkpoint.</strong></p>
    584. 

  584. <p>**-”ema_net” [Optionall, exists  if training was performed with EMA] - **</p>
    585. 

  585. <p><strong>The state dict of the EMA net.</strong></p>
    586. 

  586. <p><strong>-”optimizer_state_dict”- Optimizer’s state dict from training.</strong></p>
    587. 

  587. <p><strong>-”scaler_state_dict”- Gradient scalar state_dict from training.</strong></p>
    588. 

  588. </section>
    589. 

  589. <section id="dataset-parameters">
    590. 

  590. <h1>Dataset Parameters<a class="headerlink" href="#dataset-parameters" title="Permalink to this headline"></a></h1>
    591. 

  591. <p>dataset_params argument passed to SgModel.build_model().</p>
    592. 

  592. <p><code class="docutils literal notranslate"><span class="pre">batch_size</span></code>: int (default=64)</p>
    593. 

  593. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>Number of examples per batch for training. Large batch sizes are recommended.
    594. 

  594. </pre></div>
    595. 

  595. </div>
    596. 

  596. <p><code class="docutils literal notranslate"><span class="pre">test_batch_size</span></code>: int (default=200)</p>
    597. 

  597. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>Number of examples per batch for test/validation. Large batch sizes are recommended.
    598. 

  598. </pre></div>
    599. 

  599. </div>
    600. 

  600. <p><code class="docutils literal notranslate"><span class="pre">dataset_dir</span></code>: str (default=”./data/”)</p>
    601. 

  601. <div class="highlight-none notranslate"><div class="highlight"><pre><span></span>Directory location for the data. Data will be downloaded to this directory when received from a remote URL.
    602. 

  602. </pre></div>
    603. 

  603. </div>
    604. 

  604. <p><code class="docutils literal notranslate"><span class="pre">s3_link</span></code>: str (default=None)</p>
    605. 

  605. <p>The remote s3 link from which to download the data (optional).</p>
    606. 

  606. </section>
    607. 

  607. <section id="network-architectures">
    608. 

  608. <h1>Network Architectures<a class="headerlink" href="#network-architectures" title="Permalink to this headline"></a></h1>
    609. 

  609. <p>The following architectures are implemented in SuperGradients’ code, and can be initialized by passing their name (i.e string) to SgModel.build_model easily.</p>
    610. 

  610. <p>For example:</p>
    611. 

  611. <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">sg_model</span> <span class="o">=</span> <span class="n">SgModel</span><span class="p">(</span><span class="s2">&quot;resnet50_experiment&quot;</span><span class="p">)</span>
    612. 

  612. <span class="n">sg_model</span><span class="o">.</span><span class="n">build_model</span><span class="p">(</span><span class="n">architecture</span><span class="o">=</span><span class="s2">&quot;resnet50&quot;</span><span class="p">)</span>
    613. 

  613. </pre></div>
    614. 

  614. </div>
    615. 

  615. <p>Will initialize a resnet50 and set it to be sg_model’s network attribute, which will be used for training.</p>
    616. 

  616. <p><strong>‘resnet18’,</strong></p>
    617. 

  617. <p>** ‘resnet34’,**</p>
    618. 

  618. <p>** ‘resnet50_3343’,**</p>
    619. 

  619. <p>** ‘resnet50’,**</p>
    620. 

  620. <p>** ‘resnet101’,**</p>
    621. 

  621. <p>** ‘resnet152’,**</p>
    622. 

  622. <p>** ‘resnet18_cifar’,**</p>
    623. 

  623. <p>** ‘custom_resnet’,**</p>
    624. 

  624. <p>** ‘custom_resnet50’,**</p>
    625. 

  625. <p>** ‘custom_resnet_cifar’,**</p>
    626. 

  626. <p>** ‘custom_resnet50_cifar’,**</p>
    627. 

  627. <p>** ‘mobilenet_v2’,**</p>
    628. 

  628. <p>** ‘mobile_net_v2_135’,**</p>
    629. 

  629. <p>** ‘custom_mobilenet_v2’,**</p>
    630. 

  630. <p>** ‘mobilenet_v3_large’,**</p>
    631. 

  631. <p>** ‘mobilenet_v3_small’,**</p>
    632. 

  632. <p>** ‘mobilenet_v3_custom’,**</p>
    633. 

  633. <p>** ‘yolo_v3’,**</p>
    634. 

  634. <p>** ‘tiny_yolo_v3’,**</p>
    635. 

  635. <p>** ‘custom_densenet’,**</p>
    636. 

  636. <p>** ‘densenet121’,**</p>
    637. 

  637. <p>** ‘densenet161’,**</p>
    638. 

  638. <p>** ‘densenet169’,**</p>
    639. 

  639. <p>** ‘densenet201’,**</p>
    640. 

  640. <p>** ‘shelfnet18’,**</p>
    641. 

  641. <p>** ‘shelfnet34’,**</p>
    642. 

  642. <p>** ‘shelfnet50_3343’,**</p>
    643. 

  643. <p>** ‘shelfnet50’,**</p>
    644. 

  644. <p>** ‘shelfnet101’,**</p>
    645. 

  645. <p>** ‘shufflenet_v2_x0_5’,**</p>
    646. 

  646. <p>** ‘shufflenet_v2_x1_0’,**</p>
    647. 

  647. <p>** ‘shufflenet_v2_x1_5’,**</p>
    648. 

  648. <p>** ‘shufflenet_v2_x2_0’,**</p>
    649. 

  649. <p>** ‘shufflenet_v2_custom5’,**</p>
    650. 

  650. <p>** ‘darknet53’,**</p>
    651. 

  651. <p>** ‘csp_darknet53’,**</p>
    652. 

  652. <p>** ‘resnext50’,**</p>
    653. 

  653. <p>** ‘resnext101’,**</p>
    654. 

  654. <p>** ‘googlenet_v1’,**</p>
    655. 

  655. <p>** ‘efficientnet_b0’,**</p>
    656. 

  656. <p>** ‘efficientnet_b1’,**</p>
    657. 

  657. <p>** ‘efficientnet_b2’,**</p>
    658. 

  658. <p>** ‘efficientnet_b3’,**</p>
    659. 

  659. <p>** ‘efficientnet_b4’,**</p>
    660. 

  660. <p>** ‘efficientnet_b5’,**</p>
    661. 

  661. <p>** ‘efficientnet_b6’,**</p>
    662. 

  662. <p>** ‘efficientnet_b7’,**</p>
    663. 

  663. <p>** ‘efficientnet_b8’,**</p>
    664. 

  664. <p>** ‘efficientnet_l2’,**</p>
    665. 

  665. <p>** ‘CustomizedEfficientnet’,**</p>
    666. 

  666. <p>** ‘regnetY200’,**</p>
    667. 

  667. <p>** ‘regnetY400’,**</p>
    668. 

  668. <p>** ‘regnetY600’,**</p>
    669. 

  669. <p>** ‘regnetY800’,**</p>
    670. 

  670. <p>** ‘custom_regnet’,**</p>
    671. 

  671. <p>** ‘nas_regnet’,**</p>
    672. 

  672. <p>** ‘yolo_v5s’,**</p>
    673. 

  673. <p>** ‘yolo_v5m’,**</p>
    674. 

  674. <p>** ‘yolo_v5l’,**</p>
    675. 

  675. <p>** ‘yolo_v5x’,**</p>
    676. 

  676. <p>** ‘custom_yolov5’,**</p>
    677. 

  677. <p>** ‘ssd_mobilenet_v1’,**</p>
    678. 

  678. <p>** ‘ssd_lite_mobilenet_v2’,**</p>
    679. 

  679. <p>** ‘repvgg_a0’,**</p>
    680. 

  680. <p>** ‘repvgg_a1’,**</p>
    681. 

  681. <p>** ‘repvgg_a2’,**</p>
    682. 

  682. <p>** ‘repvgg_b0’,**</p>
    683. 

  683. <p>** ‘repvgg_b1’,**</p>
    684. 

  684. <p>** ‘repvgg_b2’,**</p>
    685. 

  685. <p>** ‘repvgg_b3’,**</p>
    686. 

  686. <p>** ‘repvgg_d2se’,**</p>
    687. 

  687. <p>** ‘repvgg_custom’**</p>
    688. 

  688. </section>
    689. 

  689. <section id="pretrained-models">
    690. 

  690. <h1>Pretrained Models<a class="headerlink" href="#pretrained-models" title="Permalink to this headline"></a></h1>
    691. 

  691. <p>Classification models</p>
    692. 

  692. <table>
    693. 

  693.   <tr>
    694. 

  694.    <td><strong>Model</strong>
    695. 

  695.    </td>
    696. 

  696.    <td><strong>Dataset</strong>
    697. 

  697.    </td>
    698. 

  698.    <td><strong>arch_params</strong>
    699. 

  699.    </td>
    700. 

  700.    <td><strong>Top-1</strong>
    701. 

  701.    </td>
    702. 

  702.    <td><strong>Latency b1 T4</strong>
    703. 

  703.    </td>
    704. 

  704.   </tr>
    705. 

  705.   <tr>
    706. 

  706.    <td>EfficientNet B0
    707. 

  707.    </td>
    708. 

  708.    <td>ImageNet
    709. 

  709.    </td>
    710. 

  710.    <td>
    711. 

  711.    </td>
    712. 

  712.    <td>77.62
    713. 

  713.    </td>
    714. 

  714.    <td>1.16ms
    715. 

  715.    </td>
    716. 

  716.   </tr>
    717. 

  717.   <tr>
    718. 

  718.    <td>RegNetY200
    719. 

  719.    </td>
    720. 

  720.    <td>ImageNet
    721. 

  721.    </td>
    722. 

  722.    <td>
    723. 

  723.    </td>
    724. 

  724.    <td>70.88
    725. 

  725.    </td>
    726. 

  726.    <td>-
    727. 

  727.    </td>
    728. 

  728.   </tr>
    729. 

  729.   <tr>
    730. 

  730.    <td>RegNetY400
    731. 

  731.    </td>
    732. 

  732.    <td>ImageNet
    733. 

  733.    </td>
    734. 

  734.    <td>
    735. 

  735.    </td>
    736. 

  736.    <td>74.74
    737. 

  737.    </td>
    738. 

  738.    <td>-
    739. 

  739.    </td>
    740. 

  740.   </tr>
    741. 

  741.   <tr>
    742. 

  742.    <td>RegNetY600
    743. 

  743.    </td>
    744. 

  744.    <td>ImageNet
    745. 

  745.    </td>
    746. 

  746.    <td>
    747. 

  747.    </td>
    748. 

  748.    <td>76.18
    749. 

  749.    </td>
    750. 

  750.    <td>-
    751. 

  751.    </td>
    752. 

  752.   </tr>
    753. 

  753.   <tr>
    754. 

  754.    <td>RegNetY800
    755. 

  755.    </td>
    756. 

  756.    <td>ImageNet
    757. 

  757.    </td>
    758. 

  758.    <td>
    759. 

  759.    </td>
    760. 

  760.    <td>77.07
    761. 

  761.    </td>
    762. 

  762.    <td>-
    763. 

  763.    </td>
    764. 

  764.   </tr>
    765. 

  765.   <tr>
    766. 

  766.    <td>ResNet18
    767. 

  767.    </td>
    768. 

  768.    <td>ImageNet
    769. 

  769.    </td>
    770. 

  770.    <td>
    771. 

  771.    </td>
    772. 

  772.    <td>70.6
    773. 

  773.    </td>
    774. 

  774.    <td>0.599ms
    775. 

  775.    </td>
    776. 

  776.   </tr>
    777. 

  777.   <tr>
    778. 

  778.    <td>ResNet34
    779. 

  779.    </td>
    780. 

  780.    <td>ImageNet
    781. 

  781.    </td>
    782. 

  782.    <td>
    783. 

  783.    </td>
    784. 

  784.    <td>74.13
    785. 

  785.    </td>
    786. 

  786.    <td>0.89ms
    787. 

  787.    </td>
    788. 

  788.   </tr>
    789. 

  789.   <tr>
    790. 

  790.    <td>ResNet50
    791. 

  791.    </td>
    792. 

  792.    <td>ImageNet
    793. 

  793.    </td>
    794. 

  794.    <td>{"pretrained_weights": "imagenet", “num_classes”:1000}
    795. 

  795.    </td>
    796. 

  796.    <td>76.3
    797. 

  797.    </td>
    798. 

  798.    <td>0.94ms
    799. 

  799.    </td>
    800. 

  800.   </tr>
    801. 

  801.   <tr>
    802. 

  802.    <td>MobileNetV3_large-150 epochs
    803. 

  803.    </td>
    804. 

  804.    <td>ImageNet
    805. 

  805.    </td>
    806. 

  806.    <td>
    807. 

  807.    </td>
    808. 

  808.    <td>73.79
    809. 

  809.    </td>
    810. 

  810.    <td>0.87ms
    811. 

  811.    </td>
    812. 

  812.   </tr>
    813. 

  813.   <tr>
    814. 

  814.    <td>MobileNetV3_large-300 epochs
    815. 

  815.    </td>
    816. 

  816.    <td>ImageNet
    817. 

  817.    </td>
    818. 

  818.    <td>
    819. 

  819.    </td>
    820. 

  820.    <td>74.52
    821. 

  821.    </td>
    822. 

  822.    <td>0.87ms
    823. 

  823.    </td>
    824. 

  824.   </tr>
    825. 

  825.   <tr>
    826. 

  826.    <td>MobileNetV3_small
    827. 

  827.    </td>
    828. 

  828.    <td>ImageNet
    829. 

  829.    </td>
    830. 

  830.    <td>
    831. 

  831.    </td>
    832. 

  832.    <td>67.45
    833. 

  833.    </td>
    834. 

  834.    <td>0.75ms
    835. 

  835.    </td>
    836. 

  836.   </tr>
    837. 

  837.   <tr>
    838. 

  838.    <td>MobileNetV2_w1
    839. 

  839.    </td>
    840. 

  840.    <td>ImageNet
    841. 

  841.    </td>
    842. 

  842.    <td>
    843. 

  843.    </td>
    844. 

  844.    <td>73.08
    845. 

  845.    </td>
    846. 

  846.    <td>0.58ms
    847. 

  847.    </td>
    848. 

  848.   </tr>
    849. 

  849. </table>
    850. 

  850. <p>Object Detection models</p>
    851. 

  851. <table>
    852. 

  852.   <tr>
    853. 

  853.    <td><strong>Model</strong>
    854. 

  854.    </td>
    855. 

  855.    <td><strong>Dataset</strong>
    856. 

  856.    </td>
    857. 

  857.    <td><strong>arch_params</strong>
    858. 

  858.    </td>
    859. 

  859.    <td><strong>mAPval</strong>
    860. 

  860. <p>
    861. 

  861. <strong>0.5:0.95</strong>
    862. 

  862.    </td>
    863. 

  863.    <td><strong>Latency b1T4</strong>
    864. 

  864.    </td>
    865. 

  865.    <td><strong>Throughout b64T4</strong>
    866. 

  866.    </td>
    867. 

  867.   </tr>
    868. 

  868.   <tr>
    869. 

  869.    <td>YOLOv5 small
    870. 

  870.    </td>
    871. 

  871.    <td>CoCo
    872. 

  872.    </td>
    873. 

  873.    <td>640x640
    874. 

  874.    </td>
    875. 

  875.    <td>37.3
    876. 

  876.    </td>
    877. 

  877.    <td>10.09ms
    878. 

  878.    </td>
    879. 

  879.    <td>101.85fps
    880. 

  880.    </td>
    881. 

  881.   </tr>
    882. 

  882.   <tr>
    883. 

  883.    <td>YOLOv5 medium
    884. 

  884.    </td>
    885. 

  885.    <td>CoCo
    886. 

  886.    </td>
    887. 

  887.    <td>640x640
    888. 

  888.    </td>
    889. 

  889.    <td>45.2
    890. 

  890.    </td>
    891. 

  891.    <td>17.55ms
    892. 

  892.    </td>
    893. 

  893.    <td>57.66fps
    894. 

  894.    </td>
    895. 

  895.   </tr>
    896. 

  896. </table>
    897. 

  897. <p>Semantic Segmentation models</p>
    898. 

  898. <table>
    899. 

  899.   <tr>
    900. 

  900.    <td><strong>Model</strong>
    901. 

  901.    </td>
    902. 

  902.    <td><strong>Dataset</strong>
    903. 

  903.    </td>
    904. 

  904.    <td><strong>arch_params</strong>
    905. 

  905.    </td>
    906. 

  906.    <td><strong>mIoU</strong>
    907. 

  907.    </td>
    908. 

  908.    <td><strong>Latency b1T4</strong>
    909. 

  909.    </td>
    910. 

  910.    <td><strong>Throughout b64T4</strong>
    911. 

  911.    </td>
    912. 

  912.   </tr>
    913. 

  913.   <tr>
    914. 

  914.    <td>DDRNet23
    915. 

  915.    </td>
    916. 

  916.    <td>Cityscapes
    917. 

  917.    </td>
    918. 

  918.    <td>
    919. 

  919.    </td>
    920. 

  920.    <td>78.65
    921. 

  921.    </td>
    922. 

  922.    <td>-
    923. 

  923.    </td>
    924. 

  924.    <td>-
    925. 

  925.    </td>
    926. 

  926.   </tr>
    927. 

  927.   <tr>
    928. 

  928.    <td>DDRNet23 slim
    929. 

  929.    </td>
    930. 

  930.    <td>Cityscapes
    931. 

  931.    </td>
    932. 

  932.    <td>
    933. 

  933.    </td>
    934. 

  934.    <td>76.6
    935. 

  935.    </td>
    936. 

  936.    <td>
    937. 

  937.    </td>
    938. 

  938.    <td>
    939. 

  939.    </td>
    940. 

  940.   </tr>
    941. 

  941. </table>
    942. 

  942. <p>Example- how to load a pretrained model:</p>
    943. 

  943. <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">sg_model</span> <span class="o">=</span> <span class="n">SgModel</span><span class="p">(</span><span class="s2">&quot;resnet50_experiment&quot;</span><span class="p">)</span>
    944. 

  944. 

  945. <span class="n">sg_model</span><span class="o">.</span><span class="n">build_model</span><span class="p">(</span><span class="n">architecture</span><span class="o">=</span><span class="s2">&quot;resnet50&quot;</span><span class="p">,</span>
    946. 

  946.                       <span class="n">arch_params</span><span class="o">=</span><span class="p">{</span><span class="s2">&quot;pretrained_weights&quot;</span><span class="p">:</span> <span class="s2">&quot;imagenet&quot;</span><span class="p">,</span> <span class="s2">&quot;num_classes&quot;</span><span class="p">:</span> <span class="mi">1000</span><span class="p">}</span>
    947. 

  947.                       <span class="p">)</span>
    948. 

  948. </pre></div>
    949. 

  949. </div>
    950. 

  950. </section>
    951. 

  951. <section id="how-to-reproduce-our-training-recipes">
    952. 

  952. <h1>How To Reproduce Our Training Recipes<a class="headerlink" href="#how-to-reproduce-our-training-recipes" title="Permalink to this headline"></a></h1>
    953. 

  953. <p>The training recipes for the pretrained models are completely visible for the SuperGradients’ users and can be found under “<em>YOUR_LOCAL_PATH_TO_SUPER_GRADIENTS_PACKAGE”/ examples/{DATASET_NAME}</em>{ARCHITECTURE_NAME}_example. _</p>
    954. 

  954. <p>_The corresponding YAML configuration files can be found under _“<em>YOUR_LOCAL_PATH_TO_SUPER_GRADIENTS_PACKAGE”/conf/{DATASET_NAME}</em>{ARCHITECTURE_NAME}_conf _</p>
    955. 

  955. <p>The configuration files include the specific instructions on how to run the training recipes for reproducibility, as well as links to our tensorboards and logs from their training. Additional information regarding training time, metric scores on different configurations can be found in the configuration files as comments as well.</p>
    956. 

  956. </section>
    957. 

  957. <section id="supergradients-faq">
    958. 

  958. <h1>SuperGradients FAQ<a class="headerlink" href="#supergradients-faq" title="Permalink to this headline"></a></h1>
    959. 

  959. <section id="what-type-of-tasks-does-the-supergradients-support">
    960. 

  960. <h2>What Type of Tasks Does the SuperGradients Support?<a class="headerlink" href="#what-type-of-tasks-does-the-supergradients-support" title="Permalink to this headline"></a></h2>
    961. 

  961. <ul class="simple">
    962. 

  962. <li><p>Classification</p></li>
    963. 

  963. <li><p>Object Detection</p></li>
    964. 

  964. <li><p>Segmentation</p></li>
    965. 

  965. </ul>
    966. 

  966. </section>
    967. 

  967. </section>
    968. 

  968. 

  969. 

  970.            </div>
    971. 

  971.           </div>
    972. 

  972.           <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
    973. 

  973.         <a href="super_gradients.training.html" class="btn btn-neutral float-left" title="Training package" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
    974. 

  974.     </div>
    975. 

  975. 

  976.   <hr/>
    977. 

  977. 

  978.   <div role="contentinfo">
    979. 

  979.     <p>&#169; Copyright 2021, SuperGradients team.</p>
    980. 

  980.   </div>
    981. 

  981. 

  982.   Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
    983. 

  983.     <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
    984. 

  984.     provided by <a href="https://readthedocs.org">Read the Docs</a>.
    985. 

  985.    
    986. 

  986. 

  987. </footer>
    988. 

  988.         </div>
    989. 

  989.       </div>
    990. 

  990.     </section>
    991. 

  991.   </div>
    992. 

  992.   <script>
    993. 

  993.       jQuery(function () {
    994. 

  994.           SphinxRtdTheme.Navigation.enable(true);
    995. 

  995.       });
    996. 

  996.   </script> 
    997. 

  997. 

  998. </body>
    999. 

  999. </html>
    1000.
Prev
Next
Tip!

Press p or to see the previous file or, n or to see the next file

Comments

Write Preview
Loading...
Integrate AWS S3
Use S3 remote
Select bucket
Access key
Finish

Use AWS S3 as storage!

Browsing data directories saved to S3 is possible with DAGsHub. Let's configure your repository to easily display your data in the context of any commit!

Specify your S3 bucket

Access key (If needed)

Congratulations!

super-gradients is now integrated with AWS S3!

Delete Storage Key

Are you sure you want to delete this access key?

No
Yes
Integrate Google Cloud Storage
Use Google Storage
Select bucket
Upload key
Finish

Use Google Cloud Storage!

Browsing data directories saved to Google Cloud Storage is possible with DAGsHub. Let's configure your repository to easily display your data in the context of any commit!

Specify your Google Storage bucket

Service Account Key

Congratulations!

super-gradients is now integrated with Google Cloud Storage!

Delete Storage Key

Are you sure you want to delete this access key?

No
Yes
Integrate Azure Cloud Storage
Use Azure Storage
Select bucket
Set key
Finish

Use Azure Cloud Storage!

Browsing data directories saved to Azure Cloud Storage is possible with DAGsHub. Let's configure your repository to easily display your data in the context of any commit!

Specify your Azure Storage bucket

Access key (If needed)

Congratulations!

super-gradients is now integrated with Azure Cloud Storage!

Delete Storage Key

Are you sure you want to delete this access key?

No
Yes
Integrate S3 compatible storage
Use S3 like remote
Select bucket
Access key
Finish

Use any S3 compatible storage!

Browsing data directories saved to S3 compatible storage is possible with DAGsHub. Let's configure your repository to easily display your data in the context of any commit!

Specify your S3 bucket

Bucket name cannot be the same as the repository name. Please change one of them.

Access key (If needed)

Congratulations!

super-gradients is now integrated with your S3 compatible storage!

Delete Storage Key

Are you sure you want to delete this access key?

No
Yes