#396 Trainer constructor cleanup

Merged
Ghost merged 1 commits into Deci-AI:master from deci-ai:feature/SG-266_clean_trainer_ctor
  
    
        
          
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

          
            <!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <title>super_gradients.training.datasets.segmentation_datasets.cityscape_segmentation &mdash; SuperGradients 1.0 documentation</title>
      <link rel="stylesheet" href="../../../../../_static/pygments.css" type="text/css" />
      <link rel="stylesheet" href="../../../../../_static/css/theme.css" type="text/css" />
      <link rel="stylesheet" href="../../../../../_static/graphviz.css" type="text/css" />
  <!--[if lt IE 9]>
    <script src="../../../../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
        <script data-url_root="../../../../../" id="documentation_options" src="../../../../../_static/documentation_options.js"></script>
        <script src="../../../../../_static/jquery.js"></script>
        <script src="../../../../../_static/underscore.js"></script>
        <script src="../../../../../_static/doctools.js"></script>
    <script src="../../../../../_static/js/theme.js"></script>
    <link rel="index" title="Index" href="../../../../../genindex.html" />
    <link rel="search" title="Search" href="../../../../../search.html" /> 
</head>
<body class="wy-body-for-nav"> 
  <div class="wy-grid-for-nav">
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search" >
            <a href="../../../../../index.html" class="icon icon-home"> SuperGradients
          </a>
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../../../../search.html" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>
        </div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
              <p class="caption"><span class="caption-text">Welcome To SuperGradients</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../../../welcome.html">Fill our 4-question quick survey! We will raffle free SuperGradients swag between those who will participate -&gt; Fill Survey</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../../welcome.html#supergradients">SuperGradients</a></li>
</ul>
<p class="caption"><span class="caption-text">Technical Documentation</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../../../super_gradients.common.html">Common package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../../super_gradients.training.html">Training package</a></li>
</ul>
<p class="caption"><span class="caption-text">User Guide</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../../../user_guide.html">What is SuperGradients?</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../../user_guide.html#introducing-the-supergradients-library">Introducing the SuperGradients library</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../../user_guide.html#installation">Installation</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../../user_guide.html#integrating-your-training-code-complete-walkthrough">Integrating Your Training Code - Complete Walkthrough</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../../user_guide.html#training-parameters">Training Parameters</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../../user_guide.html#logs-and-checkpoints">Logs and Checkpoints</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../../user_guide.html#dataset-parameters">Dataset Parameters</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../../user_guide.html#network-architectures">Network Architectures</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../../user_guide.html#pretrained-models">Pretrained Models</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../../user_guide.html#how-to-reproduce-our-training-recipes">How To Reproduce Our Training Recipes</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../../user_guide.html#professional-tools-integration">Professional Tools Integration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../../user_guide.html#supergradients-faq">SuperGradients FAQ</a></li>
</ul>
        </div>
      </div>
    </nav>
    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../../../../index.html">SuperGradients</a>
      </nav>
      <div class="wy-nav-content">
        <div class="rst-content">
          <div role="navigation" aria-label="Page navigation">
  <ul class="wy-breadcrumbs">
      <li><a href="../../../../../index.html" class="icon icon-home"></a> &raquo;</li>
          <li><a href="../../../../index.html">Module code</a> &raquo;</li>
      <li>super_gradients.training.datasets.segmentation_datasets.cityscape_segmentation</li>
      <li class="wy-breadcrumbs-aside">
      </li>
  </ul>
  <hr/>
</div>
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
             
  <h1>Source code for super_gradients.training.datasets.segmentation_datasets.cityscape_segmentation</h1><div class="highlight"><pre>
<span></span><span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">cv2</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">from</span> <span class="nn">PIL</span> <span class="kn">import</span> <span class="n">Image</span><span class="p">,</span> <span class="n">ImageColor</span>
<span class="kn">from</span> <span class="nn">super_gradients.training.datasets.segmentation_datasets.segmentation_dataset</span> <span class="kn">import</span> <span class="n">SegmentationDataSet</span>
<span class="c1"># TODO - ADD COARSE DATA - right now cityscapes dataset includes fine annotations. It&#39;s optional to use extra coarse</span>
<span class="c1">#  annotations.</span>
<span class="c1"># label for background and labels to ignore during training and evaluation.</span>
<span class="n">CITYSCAPES_IGNORE_LABEL</span> <span class="o">=</span> <span class="mi">19</span>
<div class="viewcode-block" id="CityscapesDataset"><a class="viewcode-back" href="../../../../../super_gradients.training.datasets.segmentation_datasets.html#super_gradients.training.datasets.segmentation_datasets.cityscape_segmentation.CityscapesDataset">[docs]</a><span class="k">class</span> <span class="nc">CityscapesDataset</span><span class="p">(</span><span class="n">SegmentationDataSet</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;</span>
<span class="sd">    CityscapesDataset - Segmentation Data Set Class for Cityscapes Segmentation Data Set,</span>
<span class="sd">    main resolution of dataset: (2048 x 1024).</span>
<span class="sd">    Not all the original labels are used for training and evaluation, according to cityscape paper:</span>
<span class="sd">    &quot;Classes that are too rare are excluded from our benchmark, leaving 19 classes for evaluation&quot;.</span>
<span class="sd">    For more details about the dataset labels format see:</span>
<span class="sd">    https://github.com/mcordts/cityscapesScripts/blob/master/cityscapesscripts/helpers/labels.py</span>
<span class="sd">    &quot;&quot;&quot;</span>
    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
                 <span class="n">root_dir</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
                 <span class="n">list_file</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
                 <span class="n">labels_csv_path</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
                 <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot;</span>
<span class="sd">        :param root: root directory to dataset.</span>
<span class="sd">        :param list_file: list file that contains names of images to load, line format: &lt;image_path&gt; &lt;label_path&gt;</span>
<span class="sd">        :param labels_csv_path: path to csv file, with labels metadata and mapping.</span>
<span class="sd">        :param kwargs: Any hyper params required for the dataset, i.e img_size, crop_size, cache_images</span>
<span class="sd">        &quot;&quot;&quot;</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">root_dir</span> <span class="o">=</span> <span class="n">root_dir</span>
        <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">root_dir</span><span class="p">,</span> <span class="n">list_file</span><span class="o">=</span><span class="n">list_file</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
        <span class="c1"># labels dataframe for labels metadata.</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">labels_data</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">recfromcsv</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">root_dir</span><span class="p">,</span> <span class="n">labels_csv_path</span><span class="p">),</span> <span class="n">dtype</span><span class="o">=</span><span class="s1">&#39;&lt;i8,U20,&lt;i8,&lt;i8,U12,&lt;i8,?,?,U7&#39;</span><span class="p">,</span> <span class="n">comments</span><span class="o">=</span><span class="s1">&#39;&amp;&#39;</span><span class="p">)</span>
        <span class="c1"># map vector to map ground-truth labels to train labels</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">labels_map</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">labels_data</span><span class="o">.</span><span class="n">field</span><span class="p">(</span><span class="s2">&quot;trainid&quot;</span><span class="p">)</span>
        <span class="c1"># class names</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">classes</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">labels_data</span><span class="o">.</span><span class="n">field</span><span class="p">(</span><span class="s2">&quot;name&quot;</span><span class="p">)[</span><span class="n">np</span><span class="o">.</span><span class="n">logical_not</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">labels_data</span><span class="o">.</span><span class="n">field</span><span class="p">(</span><span class="s2">&quot;ignoreineval&quot;</span><span class="p">))]</span><span class="o">.</span><span class="n">tolist</span><span class="p">()</span>
        <span class="c1"># color palette for visualization</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">train_id_color_palette</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_create_color_palette</span><span class="p">()</span>
    <span class="k">def</span> <span class="nf">_generate_samples_and_targets</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot;</span>
<span class="sd">        override _generate_samples_and_targets function, to parse list file.</span>
<span class="sd">        line format of list file: &lt;image_path&gt; &lt;label_path&gt;</span>
<span class="sd">        &quot;&quot;&quot;</span>
        <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">root_dir</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">list_file_path</span><span class="p">))</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
            <span class="n">img_list</span> <span class="o">=</span> <span class="p">[</span><span class="n">line</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">()</span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">f</span><span class="p">]</span>
        <span class="k">for</span> <span class="n">image_path</span><span class="p">,</span> <span class="n">label_path</span> <span class="ow">in</span> <span class="n">img_list</span><span class="p">:</span>
            <span class="bp">self</span><span class="o">.</span><span class="n">samples_targets_tuples_list</span><span class="o">.</span><span class="n">append</span><span class="p">((</span>
                <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">root</span><span class="p">,</span> <span class="n">image_path</span><span class="p">),</span>
                <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">root</span><span class="p">,</span> <span class="n">label_path</span><span class="p">)</span>
            <span class="p">))</span>
<div class="viewcode-block" id="CityscapesDataset.target_loader"><a class="viewcode-back" href="../../../../../super_gradients.training.datasets.segmentation_datasets.html#super_gradients.training.datasets.segmentation_datasets.cityscape_segmentation.CityscapesDataset.target_loader">[docs]</a>    <span class="k">def</span> <span class="nf">target_loader</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">label_path</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Image</span><span class="p">:</span>
        <span class="sd">&quot;&quot;&quot;</span>
<span class="sd">        Override target_loader function, load the labels mask image.</span>
<span class="sd">            :param label_path:  Path to the label image.</span>
<span class="sd">            :return:                     The mask image created from the array, with converted class labels.</span>
<span class="sd">        &quot;&quot;&quot;</span>
        <span class="c1"># assert that is a png file, other file types might alter the class labels value.</span>
        <span class="k">assert</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">splitext</span><span class="p">(</span><span class="n">label_path</span><span class="p">)[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="s2">&quot;.png&quot;</span>
        <span class="n">label</span> <span class="o">=</span> <span class="n">cv2</span><span class="o">.</span><span class="n">imread</span><span class="p">(</span><span class="n">label_path</span><span class="p">,</span> <span class="n">cv2</span><span class="o">.</span><span class="n">IMREAD_GRAYSCALE</span><span class="p">)</span>
        <span class="c1"># map ground-truth ids to train ids</span>
        <span class="n">label</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">labels_map</span><span class="p">[</span><span class="n">label</span><span class="p">]</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">uint8</span><span class="p">)</span>
        <span class="k">return</span> <span class="n">Image</span><span class="o">.</span><span class="n">fromarray</span><span class="p">(</span><span class="n">label</span><span class="p">,</span> <span class="s1">&#39;L&#39;</span><span class="p">)</span></div>
    <span class="k">def</span> <span class="nf">_create_color_palette</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot;</span>
<span class="sd">        Create color pallete for visualizing the segmentation masks</span>
<span class="sd">        :return: list of rgb color values</span>
<span class="sd">        &quot;&quot;&quot;</span>
        <span class="n">palette</span> <span class="o">=</span> <span class="p">[]</span>
        <span class="n">hex_colors</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">labels_data</span><span class="o">.</span><span class="n">field</span><span class="p">(</span><span class="s2">&quot;color&quot;</span><span class="p">)[</span><span class="n">np</span><span class="o">.</span><span class="n">logical_not</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">labels_data</span><span class="o">.</span><span class="n">field</span><span class="p">(</span><span class="s2">&quot;ignoreineval&quot;</span><span class="p">))]</span><span class="o">.</span><span class="n">tolist</span><span class="p">()</span>
        <span class="k">for</span> <span class="n">hex_color</span> <span class="ow">in</span> <span class="n">hex_colors</span><span class="p">:</span>
            <span class="n">rgb_color</span> <span class="o">=</span> <span class="n">ImageColor</span><span class="o">.</span><span class="n">getcolor</span><span class="p">(</span><span class="n">hex_color</span><span class="p">,</span> <span class="s2">&quot;RGB&quot;</span><span class="p">)</span>
            <span class="n">palette</span> <span class="o">+=</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">rgb_color</span><span class="p">]</span>
        <span class="k">return</span> <span class="n">palette</span>
<div class="viewcode-block" id="CityscapesDataset.get_train_ids_color_palette"><a class="viewcode-back" href="../../../../../super_gradients.training.datasets.segmentation_datasets.html#super_gradients.training.datasets.segmentation_datasets.cityscape_segmentation.CityscapesDataset.get_train_ids_color_palette">[docs]</a>    <span class="k">def</span> <span class="nf">get_train_ids_color_palette</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">train_id_color_palette</span></div>
<div class="viewcode-block" id="CityscapesDataset.target_transform"><a class="viewcode-back" href="../../../../../super_gradients.training.datasets.segmentation_datasets.html#super_gradients.training.datasets.segmentation_datasets.cityscape_segmentation.CityscapesDataset.target_transform">[docs]</a>    <span class="nd">@staticmethod</span>
    <span class="k">def</span> <span class="nf">target_transform</span><span class="p">(</span><span class="n">target</span><span class="p">):</span>
        <span class="sd">&quot;&quot;&quot;</span>
<span class="sd">        target_transform - Transforms the sample image</span>
<span class="sd">        This function overrides the original function from SegmentationDataSet and changes target pixels with value</span>
<span class="sd">        255 to value = CITYSCAPES_IGNORE_LABEL. This was done since current IoU metric from torchmetrics does not</span>
<span class="sd">        support such a high ignore label value (crashed on OOM)</span>
<span class="sd">            :param target: The target mask to transform</span>
<span class="sd">            :return:       The transformed target mask</span>
<span class="sd">        &quot;&quot;&quot;</span>
        <span class="n">out</span> <span class="o">=</span> <span class="n">SegmentationDataSet</span><span class="o">.</span><span class="n">target_transform</span><span class="p">(</span><span class="n">target</span><span class="p">)</span>
        <span class="n">out</span><span class="p">[</span><span class="n">out</span> <span class="o">==</span> <span class="mi">255</span><span class="p">]</span> <span class="o">=</span> <span class="n">CITYSCAPES_IGNORE_LABEL</span>
        <span class="k">return</span> <span class="n">out</span></div></div>
</pre></div>
           </div>
          </div>
          <footer>
  <hr/>
  <div role="contentinfo">
    <p>&#169; Copyright 2021, SuperGradients team.</p>
  </div>
  Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
    <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
    provided by <a href="https://readthedocs.org">Read the Docs</a>.
   
</footer>
        </div>
      </div>
    </section>
  </div>
  <script>
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script> 
</body>
</html>

          
        
      

  
Tip!
Press p or to see the previous file or, n or to see the next file
Deci-AI / super-gradients connected to https://github.com/Deci-AI/super-gradients.git

#396 Trainer constructor cleanup

Deci-AI
/
super-gradients
connected to https://github.com/Deci-AI/super-gradients.git
