Removal of the Trusted Authors Program

written by William Patton on August 20, 2019 in WordPress with no comments

As of August 13th 2019 we closed the Trusted Authors Program. With the program we intended to solve some problems faced by the Theme Review Team and by Theme Authors that release themes through the Theme Directory. Ultimately the TA program failed to meet expectations and it remained controversial throughout it’s existence.

The Team Leads, myself included, along with other senior reviewers collectively wrote and shared a post detailing how it was controversial and did not meet some expectations. The official post was not the place for me to be sharing my own personal opinions of the program but there is some of that in this post here.

The Root of the Problem We Tried To Solve

On the Theme Review Team we face a constant struggle to keep the submissions queue length down so that authors don’t have long wait times before they receive a review with potential change requests. We wanted to make the queue shorter, reviews more easy going for people.

It’s a persistent problem and is particularly bad if a theme is ultimately closed due to too having many issues. The experience can be quite negative for both the author and for the reviewer.

To try and and improve the situation the idea that certain authors could be trusted to already know and follow all the guidelines was passed around and eventually led to the creation of the Trusted Author Program.

Things that the Trusted Author Program for the Directory Hoped to achieve:

The original intention of the program was really two fold:

  1. To allow reviewers to focus time on reviews for themes that needed more work and to spend less of their time on the themes that are expected to be completely error free when uploaded.
  2. To give an avenue for theme authors that are capable of releasing high quality themes, with no errors, to do so without the need to wait the queue length for a review.

We wanted to speed things up for authors.

How We Expected the TA Program Would Help

As TA members authors would be expected to upload themes without issues and that follow all the guidelines. In return Trusted Author members would be allowed to submit themes more frequently and with minimal review.

Team Leads would handle all submissions through TA program and that would mean reviewers no longer needed to spend their time on them, this would free up a significant amount of resources in terms of reviewers available time

There Were Some Positives – It Provided a Goal for Theme Authors

The Trusted Authors Program gave authors something to aim for above the normal theme release. It encouraged them to check everything worked before they submitted. I seen authors try very hard to be uploading themes without issue and asking questions to confirm they were in line with guidelines. People trying to join the program were really pushing to improve and release error free.

I spoke with lots of the authors directly in tickets and in DMs and what I was hearing from authors was that they liked the goal of TA and were looking forward to reaching the point they could be trusted to release their themes through the program. Being a member of TA had benefits.

It was those benefits that people coveted. Frequent releases with minimal review is a stark contrast considering historically the TRT queue has been slow and reviews demanding for as long as most of us can remember.

Problems it caused:

While the TA program created a number of positive benefits and goals for theme authors it also created some negative effects as well. The effort required to operate TA was not small and at times it was a burden I did not take my share of.

  • The application process was hard to manage. Entry criteria difficult to set and uphold.
    • I struggled immensely to separate my feelings from the decisions and this was especially difficult for me to make the decisions I signed up to make.
  • Management of the queue for TAs required separate reports to be created in trac. Those reports needed to be written in with the existing SQL and to be modified for each new TA member.
    • This was not sustainable. We had already hit limits on the number of query lines we could run for each report and had moved to needing 2 exclusively for TA reports with a 3rd likely to be needed before the end of the year.
  • Being in the Trusted Authors Program had benefits unavailable for other authors.
    • How fair this is for everyone was commented on regularly.
    • The upsides for TA sometimes resulted in a downside for the other authors.
  • A high number of the releases were child themes or clone themes that did not meet minimum levels of difference to really qualify as a theme (or child theme) in it’s own right.
    • These themes provide no real benefit to users and really just clog up the directory with the same old repeated features and designs. It’s really not something I was proud to say I was setting live on a regular basis.
  • It was always the intention that themes released through the Trusted Authors Program would not need reviewed but it ended up being that a significant amount of them did need a review and those reviews almost always surfaced problems that needed resolved.
    • Needing to perform reviews – even partial reviews – removed one of the biggest benefits of this program to the team. It was no longer resulting in a large time saving. It also resulted in the non-TA queues being slowed significantly because senior reviewers were being tied up in the TA reviews and followups.
  • When some past reviewers joined TA program it seemed as though their motivation to perform reviews was gone.
    • The end result of this was that we had less reviewers around to ensure the queue progressed at a steady pace.
  • With Team Leads handling problems resulting from the TA program it took our time away from other important issues.
    • Most noticeably focus was taken away from keeping the final queue moving along fairly. It was commented on more than once.

Problems With The Set Entry Criteria

Getting into TA basically involved your most recent uploaded themes having very few issues when it reached review. Ideally zero blocking issues and it helped if the ticket gave some kind of indication of how well the author understood the reviewers requests – and by extension I expected this also indicated their understanding of the guidelines.

Team leads making the decision about who gets in and does not ended up needing reviews to be done again for each applicant. For various different reasons relying on the existing comments in the recent tickets alone was not enough. It was much more time consuming than expected to be digging deeply into each applicant.

Evaluating who could join and who could not was not as easy a decision to make as we hoped. I personally struggled to make these decisions efficiently and wound up spending more time making decisions about entry than I really should have needed to. It felt overly judgemental to be making the decisions on what is a quality theme and what is not.

Since the entry criteria was open to interpretation I, myself, interpreted it quite widely. After a while the group in TA were not all people I truly trusted and it started to defeat the purpose of it’s namesake – the trust aspect of it. The fact that there were so many people in the group that I did not trust was entirely my own fault. I allowed entry to people I was not certain could handle releasing themes totally error free.

Some of my observations:

There were a large number of themes released through the TA program over the last year but only a small number of them were really unique themes. Most of the releases were either child themes or they were parent themes that really should have been child themes. Themes that differed only by font selection and the images in the screenshot shouldn’t be acceptable – that is essentially a clone theme. There were many clone themes released.

Frequent releases pushed some authors to release as often as they could – but they were not able to sustain quality or variety with that schedule which is why I think a great number of clones were released. Those themes really have no added value for end users. Since TA themes were set live with minimal review (because they should not have needed one) clone themes were not immediately spotted and only were highlighted when an individual notices and reports it. Reactive action here is much less efficient than preemptive active.

Not everyone that became a member of TA program released themes with no issues. A lot of the time TA themes still required reviews to find and fix problems and those fixed problems recurred on several occasions in future releases. I even had to deal with themes that had PHP fatal errors resulting in the theme actually breaking sites or failing to activate at all.

The quality of themes in TA I expected to be incredibly high from the outset and remain high always. That did not become the case. On average the quality was the same between TA and non-TA releases.

One major downside to the TA program which I did not foresee at all was that the original motivation to perform reviews – so the queue and thus their own themes were able to be reviewed faster – was gone. We lost a pretty large amount of semi-regular (and experienced) reviewers this way and I still miss seeing them in the channel regularly.

What is Next for Trusted Authors

Authors that were members of the Trusted Author Program have returned to the standard queue along with the other theme authors that submit to the .org directory. Everyone will be reviewed from the same queue, in an order that is preferential based on how long someone has waited. All authors will be treated the same in this regard.

I encourage the authors that were TA members to continue to submit themes, review other themes and help out managing the queue. Without them helping the queue for everyone will quickly become unmanageable.

My Biggest Regrets

I have regrets about how the program was formed and how it was run. My biggest regret is not being able to fix TA program so that it could continue to exist. There were a small number of authors that flourished through TA program. Those authors released high quality themes when they had them ready for release. Error free, good user experience, taking care to follow and stay withing guidelines.

Those authors are the ones that TA was created to help. I am sorry that I did not find a better way to help enable those authors.

At the end of the day TA program was an experiment that I did not track well nor keep a close enough eye on. I was not able to communicate exact status – numbers of releases, % of those releases that were considered clones, how many had problems. What was clear though is that it was imbalanced. Members of the TA program – and those aiming to join – had mostly positive impressions, non-members largely negative.

It is the fact that the impressions I heard from people was largely negative that drove the decision to ultimately remove the program.