Remove temp cache table #377
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
this is completely ignorant of re-issued data, so when a new issue of data happens, the cache may be somewhat stale and/or inaccurate. the inaccuracies will probably be relatively small if the reissues are not wildly different than the previous issue. also, those differences will get washed out as data series continue to grow in size. a ~weekly full population of the cache should probably be done to clean up those inaccuracies. this will break ingestion if new data is loaded before fully populating the cache with the existing data, because it expects a new field in the cache: 'num_points'. the code is currently saving results to a 'test' cache table for comparing with the regular/main cache table. TODOs have been included for lines that need to be reverted if/when testing is successful. i have a feeling that the incremental changes could result in some loss of precision for the sd values, but i may just be paranoid. theoretically, the results should be identical if floats have infinite precision. changed 'Database.get_covidcast_meta()' to 'Database.compute_covidcast_meta()' for clarity
Co-authored-by: Katie Mazaitis <[email protected]>
(after successfully testing in prod) dont forget to follow with: DROP TABLE `covidcast_meta_cache_test`;
krivard
reviewed
Apr 10, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
(after successfully testing in prod)
dont forget to follow with:
DROP TABLE `covidcast_meta_cache_test`;#289