Adaptive Training -- How long does a beta usually last?

Sorry, working late. its 1:30am.

2 Likes

Nate also said in the announcment video (in Feb) that his system processed outside workouts, and that was coming soon

1 Like

I think a lot of the early steps were very much alpha, if not development. But as far as my team would be concerned, this wasn’t ever a beta test, and isn’t currently, and the main reason for that would be that the purpose of a beta test is to determine if it works, and not that “words are spelt correctly” and “does the survery list the right order”, but does it actually work, as in, does my FTP actually improve when following adaptive training at a greater rate than if i wasn’t, and also the users weren’t set an expectation of was expected of them during the beta test, these are prerequisits for us, but I work in software in the healt industry and mistakes aren’t as simple “ftp has gone down”, this to me (but again it could be my training), this is about throwing untested code at users, calling it a beta, and seeing what happens

For me the beta test would have been “does this improve ftp at a faster rate a standard plan” as far as I can tell, this hasn’t been tested (there is another thread about ftp going down)

3 Likes

Well everyone is different, so the test should be for the majority rather than 100%. Additionally as I mentioned above, the people posting on here are likely to be the vocal minority (I guess there must be tens of thousands of people using AT now, and how many are posting on here?), so it would be surprising if the feedback on here was totally representative.

Personally I had great results using it this year, setting all time PRs etc across all durations, and I think this was mainly due to better compliance with a smoother progression - previously in the fixed plans there used to be a few progressions in Build which would totally wipe me out, then I’d lose motivation and basically stop following the plan. But I am not making threads about it :slight_smile:

4 Likes

I totally agree, you should be very careful about analysing a forum, only the angry people post, but two things

  1. You post is a cop out, you are saying that all is well, because the thousands of people that using AT, only the angry ones are posting, which dismisses the angry ones as irrelivant, without evidence, not everybody that is angry posts either, so the best feedback is no feedback ?
  2. That wasn’t what I wasn’t what I was talking about, so you are dismissing my post with “happy users don’t post”, I was talking about the aim of a beta test would be to detemine if the product work “does it improve users ftp”, and by your own post, this hasn’t been done. because the “the happy users haven’t posted”

Feedback from a beta test should never be “how many people post in a forum”

Oh no, that wasn’t what I meant, sorry!

I was only really saying that just because a few people have threads saying they’ve had issues doesn’t invalidate the beta - no training approach will work for 100% of users, surely. I totally agree that you can’t validate a beta just because angry people aren’t posting on a forum! The evidence will be in the data, which we don’t have access to, but TR does - would be cool if they could post something, even high level.

Oh I totally agree, but I am saying (suggesting) is that this was never a beta, because it didn’t set user expectation, or reporting

I’m sorry but that is another dismissive statement, and can’t be checked or questioned, it was used on people who said that the progression rates of TR plans was to high (look in my history) and when DJ posted his video, why there was so much feedback, because so many people were dismissed

And only using your data will be miss leading (if you only look at patients who have died from cancer, 100% of people die from cancer), TR data can quickly be skewed by people not using strava, triathletes would have a different progression rate than cyclist, but TR don’t gather swimming/running data (or other sports), do TR actually employ statisticians ?

It’s just a easy comment to dismiss other people, and doesn’t fit into what should be required in settting up a beta test, and setting user expectations, and that the user behavior is consistent during the beta test (do they upload to TR)

Please understand: not very.

But it’s true isn’t it? We can’t know how effective AT is unless TR publishes some evidence, as they hold all of the data here.

Even if you were to ask people to contribute you are going to get only a subset of data (i.e., those engaged users who post in this forum, unless you invest a lot more time into it).

To whether TR employs statisticians - I would guess yes but obviously I have no better idea than anyone else who doesn’t work there.

To be clear I think they should show some evidence. At the moment we basically just have their word that people on AT do better than those who aren’t on AT.

1 Like

Maybe, but you dismissed my argument with “TR have the data”

Exactly (presuming they have actually looked at it), but , again, my post was entirely about is this a beta test, not if AT works, you dismissed that with TR have the data, in my experiance, thats no a guarrente of accuracy, and like I said, a dismissive comment that can’t be questioned as unless “TR publishes some evidence, as they hold all of the data here.”, which wouldn’t be per reviewed

And thats me out of here

That’s a fundamental limitation of this test. Unless TR decides to publish something (yes it’d be great if it was peer reviewed too) we won’t get any further information.

I wasn’t meaning to be dismissive but there is literally no better answer at the moment - it is all in TR’s hands. I certainly can’t claim to know that AT is better than any other system, all I can say with any certainty is that it worked well for me this year.

Never commented on AT working / no working

OK - I certainly can’t say whether the AT beta has been successful and nor can anyone else outside of TR. Is that better?

No, wasn’t commenting on AT Success or Success of the beta at all, in any way

After all , I am using AT

Actually you can’t. All you can say is that you “improved”. You don’t know if this improvement was due to:

  • TR making the plans easier. And even without AT, you would have improved because the easier plans didn’t over tax you as much, increased your compliance, etc.
  • Simply doing another year of training, you would have made the same, or better, improvements
  • Since TR hasn’t actually said what AT is optimizing for, you don’t know if how you are viewing “improvement” is even what AT is trying to “improve”. So for your N=1, this could all be a big coincidence
  • Placebo effect - your improved compliance / improvement could simply because you believed that AT would be beneficial, so mentally you approached all workouts with a different belief about your ability to complete the workout. And the “improvement” you are noticing is simply due to this

This is my big problem with AT at the moment: we don’t actually know how TR is evaluating its performance, and more importantly, what are they trying to optimize around? Nor to the best of my knowledge has TR done a large A/B test: Randomly assign a large number of people to one of the following groups:

  • Group A (vaguely the control group): 100% indoor rides using the TR app and new plans “as is” without AT
  • Group B (the AT group): 100% indoor rides using the TR app and new plans + AT

You could easily add more groups that would eliminate the strict need for 100% indoor rides using the TR app, but given that these would introduce “noise” / potential confounding factors, I view this as above and beyond the 2 base groups called out above that would be the most straight forward way to see if AT actually is beneficial.

You are right, it is entirely my perception. The smoother progression in Build is probably the main factor which kept me compliant, as I have got quite a bit of experience with TR plans over the years.

Theres lots of things to like about it for me - I like the PLs and easy selection of sensible alternates etc - but nothing quantitative.

1 Like

I seem to remember that when AT was in closed beta that TR were looking at bring individuals on board who fitted certain usage profiles. My assumption is that this was to do some A/B testing. Of course we’re all individuals so another assumption is that they were trying to have lots of smaller groups: TR users under one year with FTP improvements of 10 - 30% since starting; that sort of thing. Get ten or fifteen such sub-groups with around 50 in each A &B set and you can start to work out what’s working and what isn’t.

Given that many papers base their research and conclusions on fewer than twenty individuals ( a guess but I doubt it’s far out) it’s likely that TR have a better base on which to make decisions.

That’s why I was careful to say “nor to the best of my opinion”. If TR did do careful A/B testing, then this would be great. And hopefully they will talk about this, even obliquely in the Q&A next week. I would especially like them to talk about what they are optimizing for:

  • Compliance - hypothesize that increased compliance to plan leads to better “fitness” / increased “FTP” / ???
  • FTP increase - this is another whole kettle of fish, because if you used “FT” to mean 75% of Ramp Test, then AT could increase “FTP” by making people more anaerobic to do better on the RAMP test. Which wouldn’t really increase “FTP” - just ramp test value
  • ???

And then you get into a whole other issue of what timeframe are you looking over for “improvement”: this season? the next 2 seasons? ???