Register
Login
Resources
Docs Blog Datasets Glossary Case Studies Tutorials & Webinars
Product
Data Engine LLMs Platform Enterprise
Pricing Explore
Connect to our Discord channel

test_preprocessing.py 740 B

You have to be logged in to leave a comment. Sign In
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
  1. from github_search import text_preprocessing
  2. import pytest
  3. @pytest.mark.parametrize("url,expected", [
  4. ('https://github.com/mperham/sidekiq#requirements', 'xxurl #requirements'),
  5. ('https://github.com/mperham/sidekiq', 'xxurl ')
  6. ])
  7. def test_url_replacement(url, expected):
  8. rule = text_preprocessing.Rules.url_replacement
  9. cleaned_url = text_preprocessing.replace_with_rules(url, [rule])
  10. assert cleaned_url == expected
  11. @pytest.mark.parametrize("url,expected", [
  12. ('#', 'xxhashtag '),
  13. ('##', 'xxhashtag ')
  14. ])
  15. def test_hashtag_replacement(url, expected):
  16. rule = text_preprocessing.Rules.hashtag_replacement
  17. cleaned_url = text_preprocessing.replace_with_rules(url, [rule])
  18. assert cleaned_url == expected
Tip!

Press p or to see the previous file or, n or to see the next file

Comments

Loading...