ds4se/docs/5.0_experiment.mining.ir.unsupervised.w2v-exp6.html

---

title: Experimenting Neural Unsupervised Approaches for Software Information Retrieval [w2v]

keywords: fastai
sidebar: home_sidebar

summary: "Just Paper. Full Experimentation. This module is dedicated to experiment with word2vec. Consider to Copy the entire notebook for a new and separeted empirical evaluation. "
---
<!--

#################################################
### THIS FILE WAS AUTOGENERATED! DO NOT EDIT! ###
#################################################
# file to edit: nbs/5.0_experiment.mining.ir.unsupervised.w2v-exp6.ipynb
# command to build the docs after a change: nbdev_build_docs

-->

<div class="container" id="notebook-container">
    
<div class="cell border-box-sizing code_cell rendered">

</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>This copy is for Cisco purposes. It was adapted to process private github data from cisco.</p>

</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">from</span> <span class="nn">ds4se.mining.ir</span> <span class="kn">import</span> <span class="o">*</span>
</pre></div>

    </div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">from</span> <span class="nn">prg</span> <span class="kn">import</span> <span class="n">prg</span>
</pre></div>

    </div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">ds4se</span> <span class="k">as</span> <span class="nn">ds</span>
</pre></div>

    </div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
</pre></div>

    </div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">import</span> <span class="nn">logging</span>
<span class="n">logging</span><span class="o">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="nb">format</span><span class="o">=</span><span class="s1">&#39;</span><span class="si">%(asctime)s</span><span class="s1"> : </span><span class="si">%(levelname)s</span><span class="s1"> : </span><span class="si">%(message)s</span><span class="s1">&#39;</span><span class="p">,</span> <span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="o">.</span><span class="n">ERROR</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Artifacts-Similarity-with-BasicSequenceVectorization">Artifacts Similarity with BasicSequenceVectorization<a class="anchor-link" href="#Artifacts-Similarity-with-BasicSequenceVectorization">&#182;</a></h1><p>We test diferent similarities based on <a href="https://www.kdnuggets.com/2017/08/comparing-distance-measurements-python-scipy.html">blog</a> and <a href="https://www.kdnuggets.com/2019/01/comparison-text-distance-metrics.html">blog2</a></p>

</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Experients-Set-up">Experients Set-up<a class="anchor-link" href="#Experients-Set-up">&#182;</a></h2>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">path_data</span> <span class="o">=</span> <span class="s1">&#39;../dvc-ds4se/&#39;</span> <span class="c1">#dataset path</span>
</pre></div>

    </div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1">#Experiments 1.2.2 &lt;&lt;-- word2vec</span>
<span class="n">path_model_prefix</span> <span class="o">=</span> <span class="n">path_data</span><span class="o">+</span><span class="s1">&#39;models/bpe/sentencepiece/wiki_py_java_bpe_128k&#39;</span>
<span class="n">path_to_trained_model</span> <span class="o">=</span> <span class="n">path_data</span><span class="o">+</span><span class="s1">&#39;/models/wv/bpe128k/[word2vec-Java-Py-SK-500-20E-128k-1594873397.267055].model&#39;</span>
<span class="k">def</span> <span class="nf">sacp_params</span><span class="p">():</span>
        <span class="k">return</span> <span class="p">{</span>
        <span class="s2">&quot;vectorizationType&quot;</span><span class="p">:</span> <span class="n">VectorizationType</span><span class="o">.</span><span class="n">word2vec</span><span class="p">,</span>
        <span class="s2">&quot;linkType&quot;</span><span class="p">:</span> <span class="n">LinkType</span><span class="o">.</span><span class="n">issue2src</span><span class="p">,</span>
        <span class="s2">&quot;system&quot;</span><span class="p">:</span> <span class="s1">&#39;sacp-python-common&#39;</span><span class="p">,</span>
        <span class="s2">&quot;path_to_trained_model&quot;</span><span class="p">:</span> <span class="n">path_to_trained_model</span><span class="p">,</span>
        <span class="s2">&quot;source_type&quot;</span><span class="p">:</span> <span class="n">SoftwareArtifacts</span><span class="o">.</span><span class="n">PR</span><span class="o">.</span><span class="n">value</span><span class="p">,</span>
        <span class="s2">&quot;target_type&quot;</span><span class="p">:</span> <span class="n">SoftwareArtifacts</span><span class="o">.</span><span class="n">PY</span><span class="o">.</span><span class="n">value</span><span class="p">,</span>
        <span class="s2">&quot;system_path_config&quot;</span><span class="p">:</span> <span class="p">{</span>
            <span class="s2">&quot;system_path&quot;</span><span class="p">:</span> <span class="s1">&#39;/tf/data/cisco/sacp_data/[sacp-python-common-all-corpus-1609224778.517111].csv&#39;</span><span class="p">,</span>
            <span class="s2">&quot;sep&quot;</span><span class="p">:</span> <span class="s1">&#39;~&#39;</span><span class="p">,</span>
            <span class="s2">&quot;names&quot;</span><span class="p">:</span> <span class="p">[</span><span class="s1">&#39;ids&#39;</span><span class="p">,</span><span class="s1">&#39;bpe128k&#39;</span><span class="p">],</span>
            <span class="s2">&quot;prep&quot;</span><span class="p">:</span> <span class="n">Preprocessing</span><span class="o">.</span><span class="n">bpe</span>
        <span class="p">},</span>
        <span class="s2">&quot;path_mappings&quot;</span><span class="p">:</span> <span class="s2">&quot;/tf/data/cisco/sacp_data/sacp-pr-mappings.csv&quot;</span><span class="p">,</span>
        <span class="s2">&quot;saving_path&quot;</span><span class="p">:</span> <span class="n">path_data</span> <span class="o">+</span> <span class="s1">&#39;metrics/traceability/experiments1.2.x/&#39;</span><span class="p">,</span>
        <span class="s2">&quot;names&quot;</span><span class="p">:</span> <span class="p">[</span><span class="s1">&#39;Source&#39;</span><span class="p">,</span><span class="s1">&#39;Target&#39;</span><span class="p">,</span><span class="s1">&#39;Linked?&#39;</span><span class="p">],</span>
        <span class="s2">&quot;model_prefix&quot;</span><span class="p">:</span> <span class="n">path_model_prefix</span>
        <span class="p">}</span>
</pre></div>

    </div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">parameters</span> <span class="o">=</span> <span class="n">sacp_params</span><span class="p">()</span>
<span class="n">parameters</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>{&#39;vectorizationType&#39;: &lt;VectorizationType.word2vec: 1&gt;,
 &#39;linkType&#39;: &lt;LinkType.issue2src: 3&gt;,
 &#39;system&#39;: &#39;sacp-python-common&#39;,
 &#39;path_to_trained_model&#39;: &#39;../dvc-ds4se//models/wv/bpe128k/[word2vec-Java-Py-SK-500-20E-128k-1594873397.267055].model&#39;,
 &#39;source_type&#39;: &#39;pr&#39;,
 &#39;target_type&#39;: &#39;py&#39;,
 &#39;system_path_config&#39;: {&#39;system_path&#39;: &#39;/tf/data/cisco/sacp_data/[sacp-python-common-all-corpus-1609224778.517111].csv&#39;,
  &#39;sep&#39;: &#39;~&#39;,
  &#39;names&#39;: [&#39;ids&#39;, &#39;bpe128k&#39;],
  &#39;prep&#39;: &lt;Preprocessing.bpe: 2&gt;},
 &#39;path_mappings&#39;: &#39;/tf/data/cisco/sacp_data/sacp-pr-mappings.csv&#39;,
 &#39;saving_path&#39;: &#39;../dvc-ds4se/metrics/traceability/experiments1.2.x/&#39;,
 &#39;names&#39;: [&#39;Source&#39;, &#39;Target&#39;, &#39;Linked?&#39;],
 &#39;model_prefix&#39;: &#39;../dvc-ds4se/models/bpe/sentencepiece/wiki_py_java_bpe_128k&#39;}</pre>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Artifacts-Similarity-with-Word2Vec">Artifacts Similarity with Word2Vec<a class="anchor-link" href="#Artifacts-Similarity-with-Word2Vec">&#182;</a></h1>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1">#[step 1]Creating the Vectorization Class</span>
<span class="n">word2vec</span> <span class="o">=</span> <span class="n">ds</span><span class="o">.</span><span class="n">mining</span><span class="o">.</span><span class="n">ir</span><span class="o">.</span><span class="n">Word2VecSeqVect</span><span class="p">(</span> <span class="n">params</span> <span class="o">=</span> <span class="n">parameters</span><span class="p">,</span> <span class="n">logging</span> <span class="o">=</span> <span class="n">logging</span> <span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stderr output_text">
<pre>2021-01-16 15:15:12,473 : INFO : adding document #0 to Dictionary(0 unique tokens: [])
2021-01-16 15:15:12,622 : INFO : built Dictionary(3580 unique tokens: [&#39;287&#39;, &#39;29&#39;, &#39;293&#39;, &#39;4)&#39;, &#39;bom&#39;]...) from 362 documents (total 149985 corpus positions)
2021-01-16 15:15:13,272 : INFO : Ignored vocab by BPE{&#39;\t&#39;, &#39;`&#39;, &#39;^&#39;, &#39;\r\n\r\n&#39;, &#39;\r\n&#39;, &#39;γ&#39;, &#39;\r\n\r\n@&#39;, &#39;```&#39;, &#39;\r\n\r\n\r\n&#39;, &#39;@&#39;, &#39;\\&#39;}
2021-01-16 15:15:13,282 : INFO : bpe preprocessing documents, dictionary, and vocab for the test corpus
2021-01-16 15:15:13,283 : INFO : loading Word2Vec object from ../dvc-ds4se//models/wv/bpe128k/[word2vec-Java-Py-SK-500-20E-128k-1594873397.267055].model
2021-01-16 15:15:13,346 : INFO : loading wv recursively from ../dvc-ds4se//models/wv/bpe128k/[word2vec-Java-Py-SK-500-20E-128k-1594873397.267055].model.wv.* with mmap=None
2021-01-16 15:15:13,347 : INFO : loading vectors from ../dvc-ds4se//models/wv/bpe128k/[word2vec-Java-Py-SK-500-20E-128k-1594873397.267055].model.wv.vectors.npy with mmap=None
2021-01-16 15:15:13,367 : INFO : setting ignored attribute vectors_norm to None
2021-01-16 15:15:13,369 : INFO : loading vocabulary recursively from ../dvc-ds4se//models/wv/bpe128k/[word2vec-Java-Py-SK-500-20E-128k-1594873397.267055].model.vocabulary.* with mmap=None
2021-01-16 15:15:13,370 : INFO : loading trainables recursively from ../dvc-ds4se//models/wv/bpe128k/[word2vec-Java-Py-SK-500-20E-128k-1594873397.267055].model.trainables.* with mmap=None
2021-01-16 15:15:13,371 : INFO : loading syn1neg from ../dvc-ds4se//models/wv/bpe128k/[word2vec-Java-Py-SK-500-20E-128k-1594873397.267055].model.trainables.syn1neg.npy with mmap=None
2021-01-16 15:15:13,413 : INFO : setting ignored attribute cum_table to None
2021-01-16 15:15:13,415 : INFO : loaded ../dvc-ds4se//models/wv/bpe128k/[word2vec-Java-Py-SK-500-20E-128k-1594873397.267055].model
2021-01-16 15:15:13,488 : INFO : precomputing L2-norms of word weight vectors
2021-01-16 15:15:13,557 : INFO : constructing a sparse term similarity matrix using &lt;gensim.models.keyedvectors.WordEmbeddingSimilarityIndex object at 0x7fc7170e64e0&gt;
2021-01-16 15:15:13,558 : INFO : iterating over columns in dictionary order
2021-01-16 15:15:13,562 : INFO : PROGRESS: at 0.03% columns (1 / 3580, 0.027933% density, 0.027933% projected density)
2021-01-16 15:15:24,971 : INFO : PROGRESS: at 27.96% columns (1001 / 3580, 0.509160% density, 1.749005% projected density)
2021-01-16 15:15:35,040 : INFO : PROGRESS: at 55.89% columns (2001 / 3580, 0.777597% density, 1.369162% projected density)
2021-01-16 15:15:44,682 : INFO : PROGRESS: at 83.83% columns (3001 / 3580, 1.015808% density, 1.206404% projected density)
2021-01-16 15:15:49,631 : INFO : constructed a sparse term similarity matrix with 1.097687% density
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1">#[step 2]NonGroundTruth Computation</span>
<span class="n">metric_list</span> <span class="o">=</span> <span class="p">[</span><span class="n">DistanceMetric</span><span class="o">.</span><span class="n">WMD</span><span class="p">,</span><span class="n">DistanceMetric</span><span class="o">.</span><span class="n">SCM</span><span class="p">,</span><span class="n">EntropyMetric</span><span class="o">.</span><span class="n">MSI_I</span><span class="p">,</span><span class="n">EntropyMetric</span><span class="o">.</span><span class="n">MI</span><span class="p">]</span>
<span class="c1">#metric_list = [EntropyMetric.MSI_I,EntropyMetric.MI]</span>
<span class="n">word2vec</span><span class="o">.</span><span class="n">ComputeDistanceArtifacts</span><span class="p">(</span> <span class="n">sampling</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">samples</span> <span class="o">=</span> <span class="mi">100</span><span class="p">,</span> <span class="n">metric_list</span> <span class="o">=</span> <span class="n">metric_list</span> <span class="p">)</span>
<span class="n">word2vec</span><span class="o">.</span><span class="n">df_nonground_link</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">

<div class="output_subarea output_stream output_stderr output_text">
<pre>2021-01-16 15:15:49,811 : INFO : Removed 1 and 6 OOV words from document 1 and 2 (respectively).
2021-01-16 15:15:49,813 : INFO : adding document #0 to Dictionary(0 unique tokens: [])
2021-01-16 15:15:49,816 : INFO : built Dictionary(261 unique tokens: [&#39;287&#39;, &#39;29&#39;, &#39;293&#39;, &#39;4)&#39;, &#39;bom&#39;]...) from 2 documents (total 1208 corpus positions)
2021-01-16 15:15:50,005 : INFO : token count processed
2021-01-16 15:15:50,188 : INFO : frequencies processed
2021-01-16 15:27:04,818 : INFO : scalar_distribution processed
2021-01-16 15:27:04,820 : INFO : entropies processed
2021-01-16 15:27:04,822 : INFO : extropies processed
2021-01-16 15:27:04,912 : INFO : token count processed
2021-01-16 15:27:04,913 : INFO : vocab #128011
2021-01-16 15:27:04,953 : INFO : alphabet_source #128011
2021-01-16 15:27:04,981 : INFO : alphabet_target #128011
2021-01-16 15:27:05,040 : INFO : diff src2tgt #set()
2021-01-16 15:27:05,132 : INFO : diff tgt2src #set()
2021-01-16 15:49:01,200 : INFO : alphabet #128011
2021-01-16 16:00:10,461 : INFO : Computed distances or similarities (&#39;295&#39;, &#39;sacp-python-common/sacp_python_common/auth_utility.py&#39;)[[1.248352530937981, 0.44477010888626706], [0.9333877861499786, 0.066612214], [0.9182958340544896, 0.9182958340544896], [4.328599539040249, 6.786873156936814, 6.8783434490672155, 4.237129246909848, 2.549743910026966, 0.09147029213040181]]
2021-01-16 16:00:10,466 : INFO : Removed 1 and 0 OOV words from document 1 and 2 (respectively).
2021-01-16 16:00:10,478 : INFO : adding document #0 to Dictionary(0 unique tokens: [])
2021-01-16 16:00:10,480 : INFO : built Dictionary(358 unique tokens: [&#39;287&#39;, &#39;29&#39;, &#39;293&#39;, &#39;4)&#39;, &#39;bom&#39;]...) from 2 documents (total 2013 corpus positions)
2021-01-16 16:00:11,054 : INFO : token count processed
2021-01-16 16:00:11,392 : INFO : frequencies processed
2021-01-16 16:11:06,520 : INFO : scalar_distribution processed
2021-01-16 16:11:06,530 : INFO : entropies processed
2021-01-16 16:11:06,531 : INFO : extropies processed
2021-01-16 16:11:06,609 : INFO : token count processed
2021-01-16 16:11:06,622 : INFO : vocab #128011
2021-01-16 16:11:06,653 : INFO : alphabet_source #128011
2021-01-16 16:11:06,680 : INFO : alphabet_target #128011
2021-01-16 16:11:06,735 : INFO : diff src2tgt #set()
2021-01-16 16:11:06,793 : INFO : diff tgt2src #set()
2021-01-16 16:33:09,271 : INFO : alphabet #128011
2021-01-16 16:44:10,492 : INFO : Computed distances or similarities (&#39;295&#39;, &#39;sacp-python-common/sacp_python_common/bandit/bandit.py&#39;)[[1.2472553644112103, 0.44498725682739837], [0.9242691695690155, 0.07573083], [0.9182958340544896, 0.9182958340544896], [4.328599539040249, 7.022418290988289, 7.083135727621986, 4.267882102406553, 2.7545361885817368, 0.06071743663369755]]
2021-01-16 16:44:10,498 : INFO : Removed 1 and 1 OOV words from document 1 and 2 (respectively).
2021-01-16 16:44:10,510 : INFO : adding document #0 to Dictionary(0 unique tokens: [])
2021-01-16 16:44:10,512 : INFO : built Dictionary(282 unique tokens: [&#39;287&#39;, &#39;29&#39;, &#39;293&#39;, &#39;4)&#39;, &#39;bom&#39;]...) from 2 documents (total 1857 corpus positions)
2021-01-16 16:44:10,884 : INFO : token count processed
2021-01-16 16:44:11,237 : INFO : frequencies processed
2021-01-16 16:55:03,225 : INFO : scalar_distribution processed
2021-01-16 16:55:03,230 : INFO : entropies processed
2021-01-16 16:55:03,231 : INFO : extropies processed
2021-01-16 16:55:03,299 : INFO : token count processed
2021-01-16 16:55:03,312 : INFO : vocab #128011
2021-01-16 16:55:03,330 : INFO : alphabet_source #128011
2021-01-16 16:55:03,378 : INFO : alphabet_target #128011
2021-01-16 16:55:03,426 : INFO : diff src2tgt #set()
2021-01-16 16:55:03,485 : INFO : diff tgt2src #set()
2021-01-16 17:27:16,696 : INFO : Computed distances or similarities (&#39;295&#39;, &#39;sacp-python-common/sacp_python_common/bandit/banditReport.py&#39;)[[1.2498384683188444, 0.444476354227881], [0.9322489202022552, 0.06775108], [1.9182958340544893, 1.2183406773511978], [4.328599539040249, 6.442520552013249, 6.511964794825376, 4.259155296228123, 2.1833652557851266, 0.06944424281212669]]
2021-01-16 17:27:16,717 : INFO : Removed 1 and 0 OOV words from document 1 and 2 (respectively).
2021-01-16 17:27:16,718 : INFO : adding document #0 to Dictionary(0 unique tokens: [])
2021-01-16 17:27:16,719 : INFO : built Dictionary(153 unique tokens: [&#39;287&#39;, &#39;29&#39;, &#39;293&#39;, &#39;4)&#39;, &#39;bom&#39;]...) from 2 documents (total 561 corpus positions)
2021-01-16 17:27:16,879 : INFO : token count processed
2021-01-16 17:27:17,196 : INFO : frequencies processed
2021-01-16 17:37:06,483 : INFO : scalar_distribution processed
2021-01-16 17:37:06,485 : INFO : entropies processed
2021-01-16 17:37:06,486 : INFO : extropies processed
2021-01-16 17:37:06,545 : INFO : token count processed
2021-01-16 17:37:06,558 : INFO : vocab #128011
2021-01-16 17:37:06,591 : INFO : alphabet_source #128011
2021-01-16 17:37:06,615 : INFO : alphabet_target #128011
2021-01-16 17:37:06,674 : INFO : diff src2tgt #set()
2021-01-16 17:37:06,724 : INFO : diff tgt2src #set()
2021-01-16 17:56:49,072 : INFO : alphabet #128011
2021-01-16 18:06:51,318 : INFO : Computed distances or similarities (&#39;295&#39;, &#39;sacp-python-common/sacp_python_common/cave/caveCa.py&#39;)[[1.2471384784196713, 0.4450104030541378], [0.9349607676267624, 0.06503923], [0.0, 0.0], [4.328599539040249, 5.855292654715939, 6.050165914126657, 4.133726279629531, 1.7215663750864074, 0.1948732594107181]]
2021-01-16 18:06:51,322 : INFO : Removed 1 and 0 OOV words from document 1 and 2 (respectively).
2021-01-16 18:06:51,335 : INFO : adding document #0 to Dictionary(0 unique tokens: [])
2021-01-16 18:06:51,337 : INFO : built Dictionary(130 unique tokens: [&#39;287&#39;, &#39;29&#39;, &#39;293&#39;, &#39;4)&#39;, &#39;bom&#39;]...) from 2 documents (total 353 corpus positions)
2021-01-16 18:06:51,546 : INFO : token count processed
2021-01-16 18:06:51,857 : INFO : frequencies processed
2021-01-16 18:16:27,353 : INFO : scalar_distribution processed
2021-01-16 18:16:27,354 : INFO : entropies processed
2021-01-16 18:16:27,355 : INFO : extropies processed
2021-01-16 18:16:27,440 : INFO : token count processed
2021-01-16 18:16:27,451 : INFO : vocab #128011
2021-01-16 18:16:27,476 : INFO : alphabet_source #128011
2021-01-16 18:16:27,497 : INFO : alphabet_target #128011
2021-01-16 18:16:27,525 : INFO : diff src2tgt #set()
2021-01-16 18:16:27,552 : INFO : diff tgt2src #set()
2021-01-16 18:35:43,106 : INFO : alphabet #128011
2021-01-16 18:45:17,873 : INFO : Computed distances or similarities (&#39;295&#39;, &#39;sacp-python-common/sacp_python_common/cave/caveSsl.py&#39;)[[1.2564126497563755, 0.44318134810490883], [0.945445828139782, 0.05455417], [0.0, 0.0], [4.328599539040249, 5.781790887408139, 6.054206773925545, 4.056183652522844, 1.7256072348852953, 0.2724158865174058]]
2021-01-16 18:45:17,895 : INFO : Removed 1 and 0 OOV words from document 1 and 2 (respectively).
2021-01-16 18:45:17,898 : INFO : adding document #0 to Dictionary(0 unique tokens: [])
2021-01-16 18:45:17,900 : INFO : built Dictionary(232 unique tokens: [&#39;287&#39;, &#39;29&#39;, &#39;293&#39;, &#39;4)&#39;, &#39;bom&#39;]...) from 2 documents (total 1812 corpus positions)
2021-01-16 18:45:18,220 : INFO : token count processed
2021-01-16 18:45:18,506 : INFO : frequencies processed
2021-01-16 18:54:53,427 : INFO : scalar_distribution processed
2021-01-16 18:54:53,438 : INFO : entropies processed
2021-01-16 18:54:53,439 : INFO : extropies processed
2021-01-16 18:54:53,502 : INFO : token count processed
2021-01-16 18:54:53,514 : INFO : vocab #128011
2021-01-16 18:54:53,542 : INFO : alphabet_source #128011
2021-01-16 18:54:53,570 : INFO : alphabet_target #128011
2021-01-16 18:54:53,616 : INFO : diff src2tgt #set()
2021-01-16 18:54:53,667 : INFO : diff tgt2src #set()
2021-01-16 19:14:22,710 : INFO : alphabet #128011
2021-01-16 19:24:10,406 : INFO : Computed distances or similarities (&#39;295&#39;, &#39;sacp-python-common/sacp_python_common/cave/caveZap.py&#39;)[[1.2553964712265058, 0.44338102535745766], [0.938563771545887, 0.06143623], [1.4591479170272448, 1.1091703386755989], [4.328599539040249, 6.321670150661742, 6.397080411051959, 4.253189278650033, 2.068480872011709, 0.0754102603902167]]
2021-01-16 19:24:10,414 : INFO : Removed 1 and 1 OOV words from document 1 and 2 (respectively).
2021-01-16 19:24:10,415 : INFO : adding document #0 to Dictionary(0 unique tokens: [])
2021-01-16 19:24:10,417 : INFO : built Dictionary(193 unique tokens: [&#39;287&#39;, &#39;29&#39;, &#39;293&#39;, &#39;4)&#39;, &#39;bom&#39;]...) from 2 documents (total 1180 corpus positions)
2021-01-16 19:24:10,525 : INFO : token count processed
2021-01-16 19:24:10,817 : INFO : frequencies processed
2021-01-16 19:33:45,065 : INFO : scalar_distribution processed
2021-01-16 19:33:45,067 : INFO : entropies processed
2021-01-16 19:33:45,068 : INFO : extropies processed
2021-01-16 19:33:45,139 : INFO : token count processed
2021-01-16 19:33:45,141 : INFO : vocab #128011
2021-01-16 19:33:45,168 : INFO : alphabet_source #128011
2021-01-16 19:33:45,196 : INFO : alphabet_target #128011
2021-01-16 19:33:45,255 : INFO : diff src2tgt #set()
2021-01-16 19:33:45,304 : INFO : diff tgt2src #set()
2021-01-16 19:53:03,130 : INFO : alphabet #128011
2021-01-16 20:02:44,509 : INFO : Computed distances or similarities (&#39;295&#39;, &#39;sacp-python-common/sacp_python_common/csbcicd_report/aggregator.py&#39;)[[1.2567729100517449, 0.4431106007813038], [0.9385907016694546, 0.0614093], [0.8112781244591328, 0.8112781244591328], [4.328599539040249, 6.075950464980766, 6.184362509765048, 4.220187494255967, 1.8557629707247987, 0.10841204478428246]]
2021-01-16 20:02:44,519 : INFO : Removed 1 and 2 OOV words from document 1 and 2 (respectively).
2021-01-16 20:02:44,530 : INFO : adding document #0 to Dictionary(0 unique tokens: [])
2021-01-16 20:02:44,533 : INFO : built Dictionary(424 unique tokens: [&#39;287&#39;, &#39;29&#39;, &#39;293&#39;, &#39;4)&#39;, &#39;bom&#39;]...) from 2 documents (total 5315 corpus positions)
2021-01-16 20:02:45,169 : INFO : token count processed
2021-01-16 20:02:45,341 : INFO : frequencies processed
2021-01-16 20:12:23,854 : INFO : scalar_distribution processed
2021-01-16 20:12:23,866 : INFO : entropies processed
2021-01-16 20:12:23,867 : INFO : extropies processed
2021-01-16 20:12:23,940 : INFO : token count processed
2021-01-16 20:12:23,940 : INFO : vocab #128011
2021-01-16 20:12:23,967 : INFO : alphabet_source #128011
2021-01-16 20:12:23,992 : INFO : alphabet_target #128011
2021-01-16 20:12:24,034 : INFO : diff src2tgt #set()
2021-01-16 20:12:24,098 : INFO : diff tgt2src #set()
2021-01-16 20:31:32,327 : INFO : alphabet #128011
2021-01-16 20:41:14,868 : INFO : Computed distances or similarities (&#39;295&#39;, &#39;sacp-python-common/sacp_python_common/csbcicd_report/csbcicd_func.py&#39;)[[1.2382078567351642, 0.44678602882695756], [0.922771543264389, 0.07722846], [2.1556390622295662, 1.2407663947533207], [4.328599539040249, 6.821527467875317, 6.848756452649715, 4.301370554265851, 2.5201569136094655, 0.027228984774398057]]
2021-01-16 20:41:14,886 : INFO : Removed 1 and 0 OOV words from document 1 and 2 (respectively).
2021-01-16 20:41:14,897 : INFO : adding document #0 to Dictionary(0 unique tokens: [])
2021-01-16 20:41:14,899 : INFO : built Dictionary(304 unique tokens: [&#39;287&#39;, &#39;29&#39;, &#39;293&#39;, &#39;4)&#39;, &#39;bom&#39;]...) from 2 documents (total 2388 corpus positions)
2021-01-16 20:41:15,313 : INFO : token count processed
2021-01-16 20:41:15,643 : INFO : frequencies processed
2021-01-16 20:50:45,036 : INFO : scalar_distribution processed
2021-01-16 20:50:45,037 : INFO : entropies processed
2021-01-16 20:50:45,040 : INFO : extropies processed
2021-01-16 20:50:45,113 : INFO : token count processed
2021-01-16 20:50:45,115 : INFO : vocab #128011
2021-01-16 20:50:45,147 : INFO : alphabet_source #128011
2021-01-16 20:50:45,186 : INFO : alphabet_target #128011
2021-01-16 20:50:45,234 : INFO : diff src2tgt #set()
2021-01-16 20:50:45,290 : INFO : diff tgt2src #set()
2021-01-16 21:10:13,787 : INFO : alphabet #128011
2021-01-16 21:20:00,902 : INFO : Computed distances or similarities (&#39;295&#39;, &#39;sacp-python-common/sacp_python_common/csbcicd_report/csbcicdReport.py&#39;)[[1.2498712607069946, 0.44446987588337045], [0.9381374716758728, 0.06186253], [1.5, 1.1225562489182657], [4.328599539040249, 6.363229521952993, 6.4238176629687995, 4.268011398024443, 2.09521812392855, 0.06058814101580623]]
2021-01-16 21:20:00,906 : INFO : Removed 1 and 0 OOV words from document 1 and 2 (respectively).
2021-01-16 21:20:00,908 : INFO : adding document #0 to Dictionary(0 unique tokens: [])
2021-01-16 21:20:00,910 : INFO : built Dictionary(196 unique tokens: [&#39;287&#39;, &#39;29&#39;, &#39;293&#39;, &#39;4)&#39;, &#39;bom&#39;]...) from 2 documents (total 573 corpus positions)
2021-01-16 21:20:01,102 : INFO : token count processed
2021-01-16 21:20:01,380 : INFO : frequencies processed
2021-01-16 21:29:39,422 : INFO : scalar_distribution processed
2021-01-16 21:29:39,423 : INFO : entropies processed
2021-01-16 21:29:39,424 : INFO : extropies processed
2021-01-16 21:29:39,484 : INFO : token count processed
2021-01-16 21:29:39,495 : INFO : vocab #128011
2021-01-16 21:29:39,523 : INFO : alphabet_source #128011
2021-01-16 21:29:39,562 : INFO : alphabet_target #128011
2021-01-16 21:29:39,616 : INFO : diff src2tgt #set()
2021-01-16 21:29:39,653 : INFO : diff tgt2src #set()
2021-01-16 21:46:21,340 : INFO : alphabet #128011
2021-01-16 21:54:42,834 : INFO : Computed distances or similarities (&#39;295&#39;, &#39;sacp-python-common/sacp_python_common/ctsm2csdl.py&#39;)[[1.2498898920918147, 0.4444661952191176], [0.9305902123451233, 0.06940979], [0.9182958340544896, 0.9182958340544896], [4.328599539040249, 6.693181005345745, 6.845182298543749, 4.176598245842245, 2.5165827595034997, 0.15200129319800393]]
2021-01-16 21:54:42,841 : INFO : Removed 1 and 0 OOV words from document 1 and 2 (respectively).
2021-01-16 21:54:42,850 : INFO : adding document #0 to Dictionary(0 unique tokens: [])
2021-01-16 21:54:42,852 : INFO : built Dictionary(422 unique tokens: [&#39;287&#39;, &#39;29&#39;, &#39;293&#39;, &#39;4)&#39;, &#39;bom&#39;]...) from 2 documents (total 2391 corpus positions)
2021-01-16 21:54:43,494 : INFO : token count processed
2021-01-16 21:54:43,805 : INFO : frequencies processed
2021-01-16 22:02:49,227 : INFO : scalar_distribution processed
2021-01-16 22:02:49,229 : INFO : entropies processed
2021-01-16 22:02:49,231 : INFO : extropies processed
2021-01-16 22:02:49,299 : INFO : token count processed
2021-01-16 22:02:49,300 : INFO : vocab #128011
2021-01-16 22:02:49,316 : INFO : alphabet_source #128011
2021-01-16 22:02:49,332 : INFO : alphabet_target #128011
2021-01-16 22:02:49,358 : INFO : diff src2tgt #set()
2021-01-16 22:02:49,384 : INFO : diff tgt2src #set()
2021-01-16 22:19:44,918 : INFO : alphabet #128011
2021-01-16 22:28:42,545 : INFO : Computed distances or similarities (&#39;295&#39;, &#39;sacp-python-common/sacp_python_common/custom_scan/customScan.py&#39;)[[1.228505767411177, 0.4487311698374851], [0.9034347161650658, 0.096565284], [2.1556390622295662, 1.2407663947533207], [4.328599539040249, 7.199944187821455, 7.246178842948281, 4.282364883913423, 2.917579303908032, 0.046234655126826674]]
2021-01-16 22:28:42,548 : INFO : Removed 1 and 0 OOV words from document 1 and 2 (respectively).
2021-01-16 22:28:42,549 : INFO : adding document #0 to Dictionary(0 unique tokens: [])
2021-01-16 22:28:42,550 : INFO : built Dictionary(64 unique tokens: [&#39;287&#39;, &#39;29&#39;, &#39;293&#39;, &#39;4)&#39;, &#39;bom&#39;]...) from 2 documents (total 109 corpus positions)
2021-01-16 22:28:42,590 : INFO : token count processed
2021-01-16 22:28:42,759 : INFO : ---------------&gt; NO SHARED INFORMATION &lt;-------------------------
2021-01-16 22:28:42,760 : INFO : frequencies processed
2021-01-16 22:28:42,761 : INFO : FREQUENCIES NOT COMPUTED!!!&lt;--------------
2021-01-16 22:28:42,813 : INFO : token count processed
2021-01-16 22:28:42,825 : INFO : vocab #128011
2021-01-16 22:28:42,851 : INFO : alphabet_source #128011
2021-01-16 22:28:42,868 : INFO : alphabet_target #128011
2021-01-16 22:28:42,894 : INFO : diff src2tgt #set()
2021-01-16 22:28:42,929 : INFO : diff tgt2src #set()
2021-01-16 22:45:01,892 : INFO : alphabet #128011
2021-01-16 22:52:43,427 : INFO : Computed distances or similarities (&#39;295&#39;, &#39;sacp-python-common/sacp_python_common/fireException.py&#39;)[[1.2633890515804782, 0.441815338508296], [0.9516591168940067, 0.048340883], [nan, nan], [4.328599539040249, 4.991331776835099, 5.632655074171856, 3.6872762417034917, 1.3040555351316065, 0.6413232973367569]]
2021-01-16 22:52:43,431 : INFO : Removed 1 and 0 OOV words from document 1 and 2 (respectively).
2021-01-16 22:52:43,432 : INFO : adding document #0 to Dictionary(0 unique tokens: [])
2021-01-16 22:52:43,433 : INFO : built Dictionary(147 unique tokens: [&#39;287&#39;, &#39;29&#39;, &#39;293&#39;, &#39;4)&#39;, &#39;bom&#39;]...) from 2 documents (total 435 corpus positions)
2021-01-16 22:52:43,583 : INFO : token count processed
2021-01-16 22:52:43,867 : INFO : frequencies processed
2021-01-16 22:59:32,689 : INFO : scalar_distribution processed
2021-01-16 22:59:32,690 : INFO : entropies processed
2021-01-16 22:59:32,691 : INFO : extropies processed
2021-01-16 22:59:32,725 : INFO : token count processed
2021-01-16 22:59:32,730 : INFO : vocab #128011
2021-01-16 22:59:32,745 : INFO : alphabet_source #128011
2021-01-16 22:59:32,762 : INFO : alphabet_target #128011
2021-01-16 22:59:32,788 : INFO : diff src2tgt #set()
2021-01-16 22:59:32,814 : INFO : diff tgt2src #set()
2021-01-16 23:13:13,412 : INFO : alphabet #128011
2021-01-16 23:20:03,937 : INFO : Computed distances or similarities (&#39;295&#39;, &#39;sacp-python-common/sacp_python_common/gosec/gosec_display.py&#39;)[[1.2565980627302673, 0.4431449341891643], [0.940634161233902, 0.05936584], [0.0, 0.0], [4.328599539040249, 6.252206315030337, 6.456994940367824, 4.123810913702762, 2.128395401327575, 0.20478862533748732]]
2021-01-16 23:20:03,944 : INFO : Removed 1 and 0 OOV words from document 1 and 2 (respectively).
2021-01-16 23:20:03,955 : INFO : adding document #0 to Dictionary(0 unique tokens: [])
2021-01-16 23:20:03,958 : INFO : built Dictionary(361 unique tokens: [&#39;287&#39;, &#39;29&#39;, &#39;293&#39;, &#39;4)&#39;, &#39;bom&#39;]...) from 2 documents (total 2196 corpus positions)
2021-01-16 23:20:04,338 : INFO : token count processed
2021-01-16 23:20:04,497 : INFO : frequencies processed
2021-01-16 23:26:57,469 : INFO : scalar_distribution processed
2021-01-16 23:26:57,471 : INFO : entropies processed
2021-01-16 23:26:57,473 : INFO : extropies processed
2021-01-16 23:26:57,508 : INFO : token count processed
2021-01-16 23:26:57,510 : INFO : vocab #128011
2021-01-16 23:26:57,538 : INFO : alphabet_source #128011
2021-01-16 23:26:57,567 : INFO : alphabet_target #128011
2021-01-16 23:26:57,616 : INFO : diff src2tgt #set()
2021-01-16 23:26:57,652 : INFO : diff tgt2src #set()
2021-01-16 23:40:36,946 : INFO : alphabet #128011
2021-01-16 23:47:29,812 : INFO : Computed distances or similarities (&#39;295&#39;, &#39;sacp-python-common/sacp_python_common/gosec/gosec_report.py&#39;)[[1.2506171441060785, 0.44432257286353766], [0.9290515631437302, 0.07094844], [1.9182958340544893, 1.2183406773511978], [4.328599539040249, 6.847583194047557, 6.904192136386496, 4.27199059670131, 2.5755925973462466, 0.056608942338939094]]
2021-01-16 23:47:29,819 : INFO : Removed 1 and 0 OOV words from document 1 and 2 (respectively).
2021-01-16 23:47:29,821 : INFO : adding document #0 to Dictionary(0 unique tokens: [])
2021-01-16 23:47:29,824 : INFO : built Dictionary(283 unique tokens: [&#39;287&#39;, &#39;29&#39;, &#39;293&#39;, &#39;4)&#39;, &#39;bom&#39;]...) from 2 documents (total 2622 corpus positions)
2021-01-16 23:47:30,045 : INFO : token count processed
2021-01-16 23:47:30,307 : INFO : frequencies processed
2021-01-16 23:54:21,383 : INFO : scalar_distribution processed
2021-01-16 23:54:21,386 : INFO : entropies processed
2021-01-16 23:54:21,387 : INFO : extropies processed
2021-01-16 23:54:21,473 : INFO : token count processed
2021-01-16 23:54:21,474 : INFO : vocab #128011
2021-01-16 23:54:21,490 : INFO : alphabet_source #128011
2021-01-16 23:54:21,507 : INFO : alphabet_target #128011
2021-01-16 23:54:21,546 : INFO : diff src2tgt #set()
2021-01-16 23:54:21,588 : INFO : diff tgt2src #set()
2021-01-17 00:08:06,423 : INFO : alphabet #128011
2021-01-17 00:15:02,830 : INFO : Computed distances or similarities (&#39;295&#39;, &#39;sacp-python-common/sacp_python_common/harden_check/harden_func.py&#39;)[[1.2203129986725987, 0.4503869502173091], [0.9176509901881218, 0.08234901], [1.4591479170272448, 1.1091703386755989], [4.328599539040249, 6.44651664457804, 6.499999342867929, 4.275116840750361, 2.171399803827679, 0.05348269828988883]]
2021-01-17 00:15:02,836 : INFO : Removed 1 and 5 OOV words from document 1 and 2 (respectively).
2021-01-17 00:15:02,838 : INFO : adding document #0 to Dictionary(0 unique tokens: [])
2021-01-17 00:15:02,840 : INFO : built Dictionary(298 unique tokens: [&#39;287&#39;, &#39;29&#39;, &#39;293&#39;, &#39;4)&#39;, &#39;bom&#39;]...) from 2 documents (total 1283 corpus positions)
2021-01-17 00:15:03,245 : INFO : token count processed
2021-01-17 00:15:03,484 : INFO : frequencies processed
2021-01-17 00:22:15,471 : INFO : scalar_distribution processed
2021-01-17 00:22:15,482 : INFO : entropies processed
2021-01-17 00:22:15,483 : INFO : extropies processed
2021-01-17 00:22:15,541 : INFO : token count processed
2021-01-17 00:22:15,541 : INFO : vocab #128011
2021-01-17 00:22:15,572 : INFO : alphabet_source #128011
2021-01-17 00:22:15,608 : INFO : alphabet_target #128011
2021-01-17 00:22:15,645 : INFO : diff src2tgt #set()
2021-01-17 00:22:15,682 : INFO : diff tgt2src #set()
</pre>
</div>
</div>

</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">word2vec</span><span class="o">.</span><span class="n">df_nonground_link</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
</pre></div>

    </div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">word2vec</span><span class="o">.</span><span class="n">df_nonground_link</span><span class="p">[</span><span class="s1">&#39;Target&#39;</span><span class="p">][</span><span class="mi">1</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1">#[step 3]Saving Non-GroundTruth Links</span>
<span class="n">word2vec</span><span class="o">.</span><span class="n">SaveLinks</span><span class="p">()</span>
</pre></div>

    </div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1">#Loading Non-GroundTruth Links (change the timestamp with the assigned in the previous step)</span>
<span class="n">df_nonglinks</span> <span class="o">=</span> <span class="n">ds</span><span class="o">.</span><span class="n">mining</span><span class="o">.</span><span class="n">ir</span><span class="o">.</span><span class="n">LoadLinks</span><span class="p">(</span><span class="n">timestamp</span><span class="o">=</span><span class="mf">1610579170.341825</span><span class="p">,</span> <span class="n">params</span><span class="o">=</span><span class="n">parameters</span><span class="p">,</span> <span class="n">logging</span><span class="o">=</span><span class="n">logging</span><span class="p">)</span>
<span class="n">df_nonglinks</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
</pre></div>

    </div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">word2vec</span><span class="o">.</span><span class="n">df_nonground_link</span> <span class="o">=</span> <span class="n">df_nonglinks</span> <span class="c1"># Only to load links from file</span>
</pre></div>

    </div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1">#[step 4]GroundTruthMatching Testing</span>
<span class="c1">#TODO change the path for a param</span>
<span class="n">path_to_ground_truth</span> <span class="o">=</span>  <span class="n">parameters</span><span class="p">[</span><span class="s1">&#39;path_mappings&#39;</span><span class="p">]</span>
<span class="n">word2vec</span><span class="o">.</span><span class="n">MatchWithGroundTruth</span><span class="p">(</span><span class="n">path_to_ground_truth</span><span class="p">,</span> <span class="n">semeru_format</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">word2vec</span><span class="o">.</span><span class="n">df_ground_link</span>
</pre></div>

    </div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">word2vec</span><span class="o">.</span><span class="n">df_ground_link</span><span class="p">[</span><span class="n">word2vec</span><span class="o">.</span><span class="n">df_ground_link</span> <span class="p">[</span><span class="s1">&#39;Linked?&#39;</span><span class="p">]</span><span class="o">==</span><span class="mi">1</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">word2vec</span><span class="o">.</span><span class="n">df_ground_link</span><span class="p">[</span><span class="s1">&#39;Source&#39;</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="4.1-Only-SACP">4.1 Only SACP<a class="anchor-link" href="#4.1-Only-SACP">&#182;</a></h2>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1">#[step 4.1]GroundTruthMatching Testing For CISCO Mappings</span>
<span class="n">word2vec</span><span class="o">.</span><span class="n">MatchWithGroundTruth</span><span class="p">(</span><span class="n">from_mappings</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">word2vec</span><span class="o">.</span><span class="n">df_ground_link</span>
</pre></div>

    </div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1">#[optional]GroundTruth Direct Processing</span>
<span class="n">ground_links</span> <span class="o">=</span> <span class="n">word2vec</span><span class="o">.</span><span class="n">ground_truth_processing</span><span class="p">(</span><span class="n">path_to_ground_truth</span><span class="p">)</span>
<span class="n">ground_links</span> <span class="c1"># A tuple</span>
</pre></div>

    </div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1">#[step 5]Saving GroundTruth Links</span>
<span class="n">word2vec</span><span class="o">.</span><span class="n">SaveLinks</span><span class="p">(</span><span class="n">grtruth</span> <span class="o">=</span> <span class="kc">True</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="c1">#Loading Non-GroundTruth Links (change the timestamp with the assigned in the previous step)</span>
<span class="n">df_glinks</span> <span class="o">=</span> <span class="n">ds</span><span class="o">.</span><span class="n">mining</span><span class="o">.</span><span class="n">ir</span><span class="o">.</span><span class="n">LoadLinks</span><span class="p">(</span><span class="n">timestamp</span><span class="o">=</span><span class="mf">1610579318.97542</span><span class="p">,</span> <span class="n">params</span><span class="o">=</span><span class="n">parameters</span><span class="p">,</span><span class="n">grtruth</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span> <span class="n">logging</span><span class="o">=</span><span class="n">logging</span><span class="p">)</span>
<span class="n">df_glinks</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
</pre></div>

    </div>
</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Generating-Documentation">Generating Documentation<a class="anchor-link" href="#Generating-Documentation">&#182;</a></h3>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">!</span> nbdev_build_docs #&lt;-------- <span class="o">[</span>Activate when stable<span class="o">]</span>
</pre></div>

    </div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="o">!</span> nbdev_build_lib
</pre></div>

    </div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="kn">from</span> <span class="nn">nbdev.export</span> <span class="kn">import</span> <span class="n">notebook2script</span>
<span class="n">notebook2script</span><span class="p">()</span>
</pre></div>

    </div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="ch">#! pip install -e .</span>
</pre></div>

    </div>
</div>
</div>

</div>
</div>