Add copy/pasting detection

[CarolineDenis]

Fixed #2996

edit: testing instructions

[chanulee1]

How do I test this?

[maxpatiiuk]

This is likely not ready for testing yet because the front-end tests are failing - but that's the intention of the change

The change will make front-end tests fail if code duplication is detected.

As soon as code duplication issues are fixed, the tests will be pass, and thus this will be ready for merging

[maxpatiiuk]

I could see one issue with this though, not sure it's important enough to fix, but something to consider:

I made a change where back-end test are only run on back-end changes, and front-end tests are run on front-end changes.

However, the code duplication check is checking both front-end and back-end - yet is only run if the front-end changes.

The worst case is that a pull request that added code duplication on the back-end gets merged, and then front-end test will start failing for people.

The fix for this would be to modify test.yml to have a third category of tests - the tests that run always, regardless of what files where changed

[maxpatiiuk]

a nice side-effect of adding this library:
it prints a nice summary of the size of the code base - https://github.com/specify/specify7/actions/runs/4407952016/jobs/7722317168

(and it ignores the test files and third-party libraries, so the numbers are not inflated like in sp6)

Shows that front-end is 23946+ 29135+ 49773+ 483 = 103,337 lines of code
Back-end is about 18352+15+755 = 19,122

[maxpatiiuk]

All good (besides the failing tests)

[maxpatiiuk]

@melton-jason can you please take a look into the following code duplication occurrences:

Clone found (python):
 - /home/runner/work/specify7/specify7/specifyweb/workbench/upload/tomany.py [59:20 - 70:3] (11 lines, 130 tokens)
   /home/runner/work/specify7/specify7/specifyweb/workbench/upload/upload_table.py [94:19 - 105:7]

Clone found (python):
 - /home/runner/work/specify7/specify7/specifyweb/workbench/upload/tomany.py [92:9 - 111:2] (19 lines, 249 tokens)
   /home/runner/work/specify7/specify7/specifyweb/workbench/upload/upload_table.py [182:9 - 201:2]

see how easy it is to fix them (if deduplicating them is adding a lot of complexity, then we can silence the duplication check for those two occurrences)

[maxpatiiuk]

@realVinayak can you also look into resolving the following. should be relatively simple

Clone found (python):
 - /Users/maxpatiiuk/site/python/specify7/specifyweb/context/collection_resources.py [14:68 - 40:2] (26 lines, 194 tokens)
   /Users/maxpatiiuk/site/python/specify7/specifyweb/context/user_resources.py [31:89 - 57:17]

Clone found (python):
 - /Users/maxpatiiuk/site/python/specify7/specifyweb/context/collection_resources.py [55:9 - 79:13] (24 lines, 205 tokens)
   /Users/maxpatiiuk/site/python/specify7/specifyweb/context/user_resources.py [58:9 - 82:14]

Clone found (python):
 - /Users/maxpatiiuk/site/python/specify7/specifyweb/context/collection_resources.py [95:71 - 118:72] (23 lines, 194 tokens)
   /Users/maxpatiiuk/site/python/specify7/specifyweb/context/user_resources.py [97:92 - 120:101]

Clone found (python):
 - /Users/maxpatiiuk/site/python/specify7/specifyweb/context/collection_resources.py [118:72 - 136:6] (18 lines, 149 tokens)
   /Users/maxpatiiuk/site/python/specify7/specifyweb/context/collection_resources.py [39:79 - 57:6]

[melton-jason]

@melton-jason can you please take a look into the following code duplication occurrences:

Clone found (python):
 - /home/runner/work/specify7/specify7/specifyweb/workbench/upload/tomany.py [59:20 - 70:3] (11 lines, 130 tokens)
   /home/runner/work/specify7/specify7/specifyweb/workbench/upload/upload_table.py [94:19 - 105:7]

Clone found (python):
 - /home/runner/work/specify7/specify7/specifyweb/workbench/upload/tomany.py [92:9 - 111:2] (19 lines, 249 tokens)
   /home/runner/work/specify7/specify7/specifyweb/workbench/upload/upload_table.py [182:9 - 201:2]

see how easy it is to fix them (if deduplicating them is adding a lot of complexity, then we can silence the duplication check for those two occurrences)

These duplicates are related to the fact that subclasses of Named Tuples are not meant to be subclassed themselves.
More specifically, subclassing is allowed, but new attributes/data can not be added to the NamedTuple, only methods.
We have conversed about this issue before, in #3240 (comment)

In short, the backend is extensively using Named Tuples to store information about the status of an upload.
Named Tuples are convenient because it is very easy to convert between Named Tuples and JSON:

specify7/specifyweb/workbench/upload/upload_plan_schema.py

Lines 228 to 245 in 393538b

	def parse_upload_table(collection, table: Table, to_parse: Dict) -> UploadTable:
	def rel_table(key: str) -> Table:
	return datamodel.get_table_strict(table.get_relationship(key).relatedModelName)
	return UploadTable(
	name=table.django_name,
	wbcols={k: parse_column_options(v) for k,v in to_parse['wbcols'].items()},
	static=to_parse['static'],
	toOne={
	key: parse_uploadable(collection, rel_table(key), to_one)
	for key, to_one in to_parse['toOne'].items()
	},
	toMany={
	key: [parse_to_many_record(collection, rel_table(key), record) for record in to_manys]
	for key, to_manys in to_parse['toMany'].items()
	}
	)

and NamedTuples are immutable once created.

Through the process of uploading or validating a workbench upload plan, the backend 'converts' NamedTuple classes into ones that have more information or are more 'specialized', up until the actual upload/validation. (A typical example of the conversion process is UploadTable -> ScopedUploadTable -> BoundUploadTable)

Because we can not subclass a generic UploadTable in ScopedUploadTable, BoundUploadTable, etc., we have to reuse a lot of the 'generic' functionality common to all of them.

[maxpatiiuk]

@melton-jason not a problem we can leave it as is given that it is presenting issues. We have the capability to silence those errors by adding comments like this - 9c52034#diff-c8cfcd0e13a828573c0d195ee26ce1cbb513e3554e37dcb0860e9053ffa16568R73-R121

Though, the issues in #3149 (comment) would be easy to fix so should be fixed

[CarolineDenis]

@melton-jason do we want to merge this at some point?

[melton-jason]

Yes, we can look into merging this.
We should probably silence the copy/paste detection as a result of #3149 (comment) and look into the errors from
#3149 (comment)

[carlosmbe]

Is this a feature that requires any UX Testing? @CarolineDenis

[CarolineDenis]

@carlosmbe , no it does not

[CarolineDenis]

@specify/dev-testing
What do we want to do with this PR?

[maxpatiiuk]

Would love to see this merged.
If you need help, I can work on this this Saturday

[realVinayak]

regarding backend workbench, is there a way to silence the workbench ones for now? They should be fixed in this PR - I reworked the matching, and deleted everything from the tomany.py file 😄 (made the logic completely generic, no need for that file anymore, see diff).

[maxpatiiuk]

I used to be against OS or editor specific gitignore patterns, and was recommending using global gitignore for that but I changed my mind.

Reasons:

• Setting up global gitignore is added friction, especially for new team members or external contributors.
• Automated tools (like jscpd, eslint, prettier) and some IDEs do not look into global gitignore file

[maxpatiiuk]

I just realized that jscpd will also help us detect incorrect merge resolutions (i.e cases when a deleted file got incorrectly "resurected" during merging). Pretty nice

[maxpatiiuk]

Testing instructions:

• Ensure the Form Editor/App Resource buttons at the top of the app resource still work and look good
• Ensure the Table Format/Table Aggregation buttons at the top of a "Record Formaters" visual editor still look good
• Open User Tools -> Operations API and make sure the documentation for the /context/user_resource/* and /context/collection_resource/* endpoints looks similar or better to prodcution

[maxpatiiuk]

Re-run npx tsc --extendedDiagnostics several times with and without this change
seems like this change makes typescript type-checking of entire project faster between 0s and 1s (possibly because TypeScript can cache the resolved type?)

[maxpatiiuk]

regarding backend workbench, is there a way to silence the workbench ones for now? They should be fixed in this #4929 - I reworked the matching, and deleted everything from the tomany.py file 😄 (made the logic completely generic, no need for that file anymore, see diff).

Sounds good. Leaving these to you

[CarolineDenis]

@maxpatiiuk,
Do you know if there is a possibility to have this ran when the dev runs a command locally instead of having it as failing test / passing test in our actions?

[maxpatiiuk]

You can run the check locally at any point by calling this command from the js_src folder:

npx tsx lib/tests/jscpd.ts

We could have it run automatically in a pre-commit hook, however the command takes 11s to check entire specify7 repository, so this would slow down commits considerably.

I know that IntelliJ IDEs had this built-in live in the editor. Maybe there is a VS Code plugin for that too - but I imagine it would slow down the IDE a bit.

[pashiav]

• Verify that the Tabs buttons in app resources still look good after the change
• Verify that the "Table Format"/"Table Aggregation" tabs in the
• Go to User Tools -> Operations API -> Verify that the documentation for the /context/user_resource/* and /context/collection_resource/* endpoints looks good (or better) than production

---

Looks good so far! For the "Table Format"/"Table Aggregation" tabs, what exactly should we be focusing on testing here?

[maxpatiiuk]

ups! looks like I lost my train of thought when I was typing that 😆
I meant to say that they are supposed to look identical on this branch to how they do on production branch