Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix SCTs that fail check-object-on-root verification #197

Open
filipsch opened this issue Sep 12, 2018 · 0 comments
Open

Fix SCTs that fail check-object-on-root verification #197

filipsch opened this issue Sep 12, 2018 · 0 comments

Comments

@filipsch
Copy link
Contributor

filipsch commented Sep 12, 2018

Since testwhat v4.7.2, different SCT functions do some assertions and verifications to figure out whether the function is appropriately used. You can see it in the CHANGELOG here.

One of these assertions happens in check_object(), to check whether it is being called on the root state. This assertion is only done for courses whose course image contains the TESTWHAT_V2_ONLY env variable (more recent courses), because otherwise they would break 71 exercises that contain SCTs that violate this rule:

   course_id chapter_id     id number                                                       title
1        639       1521   9772      2                                               R code chunks
2        639       1521   9773      3                                     Customize R code chunks
3        639       1521   9775      5                                               Inline R code
4        682       1640  11701      4                                        Making a scree plot!
5       2399      10069 114116     10                        Relative change in demographic trend
6       3022       8205 172631     11                                   Generate a grid of models
7       3022       8205 172632     12                                           Evaluate the grid
8       3022       8207 172656     12                       Tuning a Random Forest via tree depth
9       3026       8217  99950     12                                               Rate-limiting
10      3851      11371 135814     14 Evaluate a modeling procedure using n-fold cross-validation
11      3851      11372 140833      6                                 Modeling an interaction (2)
12      3851      11372 140834     13                    Input transforms: the "hockey stick" (2)
13      5798      18624 214577      2                                            Create value box
14      5798      18624 214578      3                                                Create gauge
15      5798      18624 214579      4                                            Color indicators
16      5798      18624 214580      5                                                     Linking
17      5798      18624 214582      7                                                Static table
18      5798      18624 214583      8                                          Web-friendly table
19      5798      18624 214584      9                                               DT::datatable
20      5798      18625 214598      7                               Making the Dataframe Reactive
21      5798      18625 315666     13                                 Create Global Input Sidebar
22      5951      19124 220845      1                      Background on modeling for explanation
23      5951      19124 220846      2                            Exploratory visualization of age
24      5951      19124 220852      7                        The modeling problem for explanation
25      5951      19125 220856      1                          Explaining teaching score with age
26      5951      19125 220860      4                         Predicting teaching score using age
27      5951      19125 220863      7                       Explaining teaching score with gender
28      5951      19125 220868     10                      Predicting teaching score using gender
29      5951      19126 220869      1                     Explaining house price with year & size
30      5951      19126 220876      7                Explaining house price with size & condition
31      5951      19127 220879      1                              Model selection and assessment
32      5951      19127 220882      4                          Assessing model fit with R-squared
33      5951      19127 220885      7                             Assessing predictions with RMSE
34      5951      19127 220891     10                         Validation set prediction framework
35      5951      19125 273100     11                               Making predictions using rank
36      5951      19127 273113     13                         Conclusion - Where to go from here?
37      5951      19126 298464     10               Predicting house price using size & condition
38      5951      19126 298470      6                                      Interpreting residuals
39      5951      19127 298652      6                       Comparing the R-squared of two models
40      5951      19126 328548      4                    Predicting house price using year & size
41      5951      19127 342938      5                          Computing the R-squared of a model
42      5951      19127 343228      8                         Computing the MSE & RMSE of a model
43      5951      19127 343235      9                            Comparing the RMSE of two models
44      5951      19127 343408     11                              Fitting model to training data
45      5951      19127 343411     12                                     Predicting on test data
46      5951      19124 353946     10                         The modeling problem for prediction
47      5951      19124 354701      4                       Background on modeling for prediction
48      5951      19124 369666      3                                  Numerical summaries of age
49      5951      19124 369852      5                     Exploratory visualization of house size
50      5951      19124 369855      6                          Log10 transformation of house size
51      5951      19124 370102      8           EDA of relationship of teaching & "beauty" scores
52      5951      19124 370105      9            Correlation between teaching and "beauty" scores
53      5951      19124 370252     11           EDA of relationship of house price and waterfront
54      5951      19124 370257     12                      Predicting house price with waterfront
55      5951      19125 370853      2                   Plotting a "best-fitting" regression line
56      5951      19125 370855      3                     Fitting a regression with a numerical x
57      5951      19125 371740      5                     Making predictions using "beauty score"
58      5951      19125 371744      6               Computing fitted/predicted values & residuals
59      5951      19125 372155      8                       EDA of relationship of score and rank
60      5951      19125 372162      9                   Fitting a regression with a categorical x
61      5951      19125 373846     12                   Visualizing the distribution of residuals
62      5951      19127 374711      2                         Refresher: sum of squared residuals
63      5951      19127 374714      3                                      Which model to select?
64      5951      19126 374757      2                                         EDA of relationship
65      5951      19126 374760      3                                        Fitting a regression
66      5951      19126 374968      5                  Making predictions using size and bedrooms
67      5951      19126 375305     11                Making predictions using size and waterfront
68      5951      19126 375306     12                      Automating predictions on "new" houses
69      5951      19126 375357      8                                       Parallel slopes model
70      5951      19126 375359      9                      Interpreting the parallel slopes model
71      6617      28953 421290     10                                         Apply the two steps

This issue is a reminder that we have to rewrite the SCTs for these exercises because they are smelly. As soon as that is done, we can remove this v2_only() check.

How I came up with this list

  • Make a branch on testwhat that does not do the v2 only check
  • Install this branch of testwhat in docker-r-shared and tag it (with a release candidate format, as it is never the intention to deploy)
  • This makes the validator go through all exercises with this hypothetical shared image, and spit out the exercises where it goes wrong.
  • Finally, you can use this script to programmatically a list of exercise ids from the validator endpoint and their corresponding chapters and courses.
@filipsch filipsch changed the title Fix SCT that fail check-object-on-root verification Fix SCTs that fail check-object-on-root verification Sep 12, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant