Skip to content

Conversation

@dmbates
Copy link
Contributor

@dmbates dmbates commented Oct 12, 2022

  • Re-ran the movie ratings models with rectangular full packed (RFP) storage of L[2,2].
  • Added figures and discussion of the impact of RFP
  • Deleted a vestigial paragraph at the end of largescaleobserved.qmd
  • This version is more for discussion and to see where we should go than it is a final version.
  • Should we incorporate the movie classifications (Comedy, Drama, etc.) as fixed-effects?

@dmbates dmbates marked this pull request as draft October 12, 2022 19:32
@dmbates dmbates requested a review from palday October 12, 2022 19:37
using MixedModelsMakie
using ProgressMeter
using SparseArrays
#using ProgressMeter
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did you want all this commented out?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was trying to trim packages that were included in using statements but not actually used. That is a task that is probably best left until after the content and text have stabilized. For the time being I plan to comment out the using statements for packages because that is an action that can easily be reversed.

The time to evaluate the log-likelihood once goes from about 5 minutes for the square storage to a little over 6 minutes for the rectangular storage on the `mc == 1` models.
Because the number of evaluations to convergence stays the same on this set of models the time to fit the model goes from roughly 2 hours (square storage) to about 2.5 hours (rectangular storage).

As the threshold on the number of ratings for inclusion of a movie increases the relative change in storage sizes decreases as does the relative change in time to fit the model.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

Is this because the [2,1] block is a smaller fraction of the total memory use?

@palday
Copy link
Member

palday commented Oct 13, 2022

@dmbates I think it makes sense to add some fixed effects. It might even be a nice way to show that the random effects are by far the biggest computational burden -- the fixed effects aren't free, but they're comparatively cheap.

dmbates and others added 2 commits October 13, 2022 10:22
Co-authored-by: Phillip Alday <palday@users.noreply.github.com>
Co-authored-by: Phillip Alday <palday@users.noreply.github.com>
@dmbates
Copy link
Contributor Author

dmbates commented Oct 13, 2022

Can we leave this as a draft for a couple of days? I would like to make one more attempt at speeding up rankUpdate! with RFP and sparse.

@palday
Copy link
Member

palday commented Oct 13, 2022

Can we leave this as a draft for a couple of days?

@dmbates fine with me -- no rush from my side

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants