rodo's post few weeks ago prompted me to finally begin typing out some of the issues around OTW and AO3 I’ve been thinking on. The topic of how to fix OTW in terms of maintaining the volunteers it has and attracting more has been brought to forefront during last election, and a lot of thoughtful ideas have been going around focusing on different aspects of the org before and since. Apparently, even the board has finally begun their own conversation, even if it is behind closed doors for now.
Out of the discussions that are visible at least, some I agree with, others I don’t. One thing that is constant throughout all of these however, is AD&T (Accessibility, Design and Technology) committee which is currently the main decision point for almost everything related to AO3. Solutions are formed around its existing makeup, maintaining its fixed presence even though it sits at the heart of a cluster of structural issues. I think it is time to voice it however, that AD&T itself needs to be reformed and reorganized.
So rewind, who am I and what are my qualifications? My name is Eylul and I am a staffer at the Organization for Transformative Works. I joined OTW toward the end of 2009, shortly after the open beta, as a design and testing volunteer. I was part of the support subcommittee (before it became its own committee) and was involved in writing of its original training manual, I was the lead of the QA (Quality Assurance) subcommittee (which is also informally known as testing subcommittee) for 2010, and then I&O (Internationalization and Outreach) liaison of AD&T in 2011 before taking a break from the committee and nearly decided to quit the OTW altogether. I am currently an I&O staffer, and involved in the Survey and Category Change workgroups. Outside the organization I am a Fine Arts masters student starting her thesis. However I actually completed my undergraduate degree with double majors in Computer Science and Media Arts and Sciences, and done grad course work both in Computer Science and Arts. While I have made a career change away from tech field in general, I have work experience in customer support, and research experience in HCI (Human Computer Interaction).
What is the context? AD&T is short for Accessibility Design and Technology. From the name, it can be inferred that at the inception of the organization, it was meant to support many projects of the org that might have various technical needs. Except somewhere along the way AD&T became the AO3 committee in practice, solely limited to development of one project and none other. AD&T is also currently responsible for non technical aspects and issues of AO3. Decisions on strategic questions such as which features should be prioritized takes place within AD&T, even though such decisions would ideally take in more points of view. AD&T also coordinates decision-making on AO3 matters when other committees are involved. Not only all this is a huge burden on a single committee but it causes considerable conflict of interest between supporting volunteers of different roles, supporting users of the project and supporting OTW’s goals.
As it stands AD&T committee, which is almost exclusively formed of coders, wears several hats ranging from being the design process itself, to overseeing the AO3 project management, to supporting coders and testers, and to coordinating inter-committee issues related to the archive.
What do I propose? I think that the management of the AO3 as a software project, and management of technical resources of OTW needs to be completely overhauled, and propose following two changes:
- Dividing the current AD&T into 4 committees:
All of these will serve the whole OTW as needed instead of just AO3, similar to the way Systems and Translation committees currently work, although it is likely that some volunteers at least will have specific projects they prefer to focus on.
- Front-End Coding
- Back-End Coding
- To form an AO3 committee that will make the decisions on archive priorities and production that is based on advice and input from all parties, and coordinating the work from various committees.
Below, under the cut I will explain my reasoning for this proposal under several sections: On first two sections, I will explain how separating committees based on skill resources from those focused on projects can benefit OTW and AO3. Then on the following three sections I will focus on four skill based committees, and how volunteers,the AO3 and the OTW will benefit from this new structure. Then I conclude with why this restructuring becomes more and more necessary as the AO3 matures.
Accessibility, Design and Technology to All (Projects of OTW)
Currently we have several projects within the org that are in need of technical assistance. The website has Webmasters, but for example the wiki forums were, until recently, actively seeking a coder. This is a crucial project since it is in part a test case for OTW wide forums, something the board member julia_beck has been pushing for a long time now. In another case, Vidding’s Dark Archive and Torrent of Our Own cannot get off the ground in part because it is not only coding that needs to be done, but also the support structure around the project for the volunteers.
In short, we are reinventing the wheel of training, organization, and management for each starting project. Not only that but each project is intended to have a dedicated QA team, Design team, Support, etc. This is a huge overhead for an organization of OTW’s scale. Considering how long it took AO3 to establish some of these procedures, it will be a huge loss of time spent in inefficient procedures until they can be polished. Another drawback of this issue is that often these start-up, or short term projects have few technical volunteers in them. Keeping them to their own cluster takes away existing resources as training materials, mentoring, and brainstorming within a team.
I am not saying technical volunteers won’t specialize in projects. Some people will want to specialize. However if they do not, they will have the same structure backing them regardless of which project they decide to put their energy to. For the organization this will bring flexibility because it will be easier to start up new projects and training new people for them, and volunteers will still be able ask advice to the rest of the pool, and request temporary help or mentorship.
A note here. I know there are people who feel AO3 should be the only focus of OTW and that no resources should be wasted elsewhere. If you feel this way, you might feel this would just mean more resources taken away from the production of the archive. I would expect a good portion of the volunteers will still be working on AO3 for now. What is offered here is to divide the management work of the current AD&T into specializations, then allow some of these specializations to be also available to rest of the OTW projects. This division of work will actually also benefit AO3: the rest of this long post focuses on how having 5 committees instead of one will benefit the project.
It is also important to note that the OTW projects are an ecosystem. OTW Forums will allow the AO3 related feedback, like all other discussion, to happen in a space where it can more directly affect the outcome. The Dark Archive and related projects are directly related to vidders using the AO3 to link and showcase their vids. The examples can go on.
Archive Committee Rather Than AD&T: Dedicated Project Management
As I briefly mentioned above, most of the decision making on what to prioritize to code are made by coders (and testers to a lot lesser extent), since they are the ones who form the AD&T committee. It is true that requests for advice and delegations to other committees occur thanks in part to AD&T members taking the initiative. It is usually limited however to content or to design of features that will be directly used by another committee, such as tag wrangling. It is also true that there is a roadmap in place. However, how much the roadmap is put into consideration if at all, and what considerations were made when it was decided, are questions to ask.
Thus, while the status quo sort of worked enough to not burn AO3 down into cinders, it resulted in some pretty bad prioritization errors. Today, AO3 still doesn’t have a translation interface which loses us non-english speaking fans. The support board is an interface that will replace the current form with an interface that will allow users to help each other, and allow on better dialogue on requests of bug fixes and features. This feature which was proposed over a year ago, which will benefit not just the support team but the whole archive development, is also still absent. Both are pretty critical projects. Similar prioritization errors are not limited to major new features. Fixes and improvements ends up being prioritized based on if AD&T members notice its effects directly in their circle or are themselves affected by it. The problem is not the faulty judgement of individuals here, it is that it is not, and should not be, coders’ job to have the final say in what needs to be worked on most urgently. It results in a conflict of interest.
There are 3 main issues here
- As much as coders and testing lead (AD&T recruits staff most heavily from coders, followed by testers; it's relatively rare for someone to be in the committee unless they're a coder or were staff elsewhere in the org.) will try to take initiative to seek output, they will hit bias. Currently, when it is not 502 errors or a similar urgent issue on the site, coder preferences influence strongly what new features come to the archive, rather than roadmap or goals of outreach or any other reason.
- Coders and testers being actively used at managing work is a huge waste of the scarce resource that is our technical volunteer pool. It is also a bad waste of our existing volunteers who are talented in coordinating and managing.
- That AD&T becomes the key point of decision causes a lot of interest in being in the committee. This can result in a lot of people at a time who would rather do what they are in the org to do, rather than talk about topics that are not their interest OR expertise area but sticking with it because they want to have a say. It also results in people on other committees who should probably pitch in to not have a say in the final decision.
Design as its Own Committee
The current design process of AO3 is a combination of two approaches. In some cases, a coder just produces a feature, with little to no planning or documentation and showing it around in next AD&T where there is time. Other times, a design is discussed in the AD&T committee itself.
The problems with this process are several
- Since there is no design process,the decisions on new features are often made by discussing them in the committee. AD&T tries to alleviate this through their own initiative, looking to see if there are support requests for features. However, often if a senior coder wants something in, and are willing to code it themselves, unless a drawback is obvious, it will happen. There are no user models, no target audience documented. Nobody really thinks what the workflow of a user will be until the work is nearly complete. This contributes to the prioritization issues mentioned above and also results in UIs such as the Post new form, or collections form that require videos to explain.
- If you want to design, you need to be part of the AD&T committee, which implies unless you have been designated as a QA lead or Design lead, or another position by the committee, you need to be a coder. This closes the doors on people who wish to focus on usability or aesthetics and especially those who specializes in them on expert level, who might not have the time to commit to be a coder in expected time commitment AND design.
- AD&T assumes the role of design committee itself. As mentioned above, the committee does tons of other work, varying from discussing code optimization, to coordinating content changes that comes from the workgroups, to release management, and to project management. Putting everything aside to discuss a set of icons half an hour in a meeting, or spending hours of time on it over emails is a very inefficient way to do work. Brainstorming is necessary, but it is the job of design volunteers. Not of coders, not of testers, not of higher level AO3 management.
- Design by coding is a waste of scarce resources of coders. Essentially, design is a process where you want to go in several iterations going from quick to slow, adding more details and fleshing out at every step while making sure you are on right track. The designers start with a workflow, then UI, the testing beginning as against a user model, then gradually involving active test users. In situations where the design team can interact in physical space these first iterations are actually done by sketching on paper. While we have to adapt some of these early stages to our setup (the organization doesn’t meet in person), we can have the iterations, brainstorming, acquiring feedback at several stages during design, and coding from the user base. Thus by the time it gets to the backend coders they know what they are coding, and half of the code doesn’t suddenly get obsolete midway through because of settings not reflecting the common uses of the target audience, or a problem of a similar nature.
An independent design committee combined with an AO3 overseeing committee will allow solidifying the use of design process within the archive. The description I gave above mostly exemplified usability design, but visual design, illustration, data visualization are also areas we have/can use people in. Making sure we have work to do for these people, and provide them support so that they can do what they do will benefit the archive.
Generalizing the design team will benefit volunteers doing such work in all projects. Especially those working on OTW Website or Fanlore. Also some aspects of design are required on less apparent places, like graphic design in case of production of print media and items (print media, donation gifts, etc) or data visualization in case of Survey workgroup, or Communications when they need to put out statistics of any kind up. Volunteers working in these spots can also benefit from the existence of a design team with its mentoring, training and support resources.
QA (Quality Assurance) as its Own Committee
The QA’s role is to make sure that the outgoing code is truly ready for the users. It is not only sufficient that it is bug free but is also that it is of sufficient quality, in terms of functionality and design. AO3’s current QA process is realized by a subcommittee of AD&T. This subcommittee is generally known, both within and outside the org, as the testing subcommittee, despite it being clear from older documents that they were intended work as a QA team. A subcommittee within the org, as well as outside is generally known as testing committee, despite it being clear from original documents that they were intended for former. (The difference between QA and testing is out of the scope of this post, and it is sufficient to say, at the current scale of the AO3 and the org, it is perfectly acceptable for one committee and set of volunteers to have a workflow that covers both roles)
Again several problem points.
- Testers being on pressure to test code that is delayed already on a strict deadline. Therein lies one of the biggest reasons for testers attrition. The monthly release schedule and automatic test suites helped somewhat but not enough. The workload can be described as long breaks where nothing happens punctuated by days of allnighters trying to finish everything to a strict deadline.
This can be in part alleviated by TDD (Test Driven Development). This essentially means coders writing a series of automated tests before beginning to code, based on a design and documentation. jennyst who is a board member and a former AD&T staffer, has a post about this process. It doesn’t stop the need for manual QA, but it does lessen the risks of all sorts of small bugs and mistakes making to the test release. It also has the added benefit of having documentation in form of test spec which is crucial for manual testing process to be healthy. (more on this in the next item)
- Lack of documentation, which ties into the design issue. You can’t just code something large on the fly. The consequences of this for one is that often it is not clear what the QA team needs to test, for the coders to have no clear idea what they are coding in first place. In AD&T process, testing volunteers need to often ask coders what they meant to do which is both problematic in terms of time requirement to have the coder present answering questions, and it brings the bias of the coder’s point of view of the function or feature to the QA volunteer, making it harder for the QA volunteer to catch buggy cases, inconsistencies or other issues from an independent point of view.
- Lack of QA’s input in design and usability bugs. Just because something is working doesn’t mean it isn’t convoluted. Since there is no process to work on this, usually QA volunteers stumble into these. However there is no structure to bring this feedback back into the system in a way that will result in a fix other than the QA volunteer or QA lead personally convincing the coder that there is an issue.
One of the biggest problems in AD&T is that like design, QA remains an afterthought, rather than being truly part the process. It is not very surprising knowing programming culture in general and the make-up of the committee. Splitting the QA into a separate committee and having them answer a separate managing committee rather than coders, whose priority is naturally seeing their code released as quickly as possible, will help to solve a conflict of interest that arises not only between QA and coding teams, but also between coding teams and users, who might be affected by a bug or low quality update.
Separating QA also allows us to build a structure within the OTW where the team can serve other projects. Website, wiki, new projects that the organization is planning are all cases where QA is needed. For example during a recent upgrade by Systems on several databases, volunteers doing the QA for other projects to ensure nothing was broken had much less resources (docs, training, etc) to work with. As I mentioned above, the OTW projects are an ecosystem. Making sure we have a set of projects benefiting each other, and adhering to a standard of quality will return as a benefit to the AO3.
Back-End Coding and Front-End Coding: Reforming Our Coding Process
In the current structure of AD&T, coders all answer directly to the AD&T committee and are organized by them. There is no coding subcomittee. Coding volunteers are divided into two groups based on skillset.
- Back-end coders, who mainly work with ruby, functionality of features, database access, performance etc.
- beginner coders takes smaller, easy to work on bugs
- intermediate coders takes over small features (with a mentor if possible) or bigger trickier bugs.
- senior coders takes over whole features by themselves, and pick up everything else anyone in above 2 categories that cannot do on their own.
An obvious question is why is the work not shared. This is not possible when there is no set design and essentially one feels they need to take over the whole the project and go with it because of the misconception that anything else will take too long and that it is the only way their vision on how the feature should be will be realized. Furthermore at least some of our coders, having no experience in working as a team on same code, are probably not aware of how to use and write documentation to collaboratively code, and thus in general feel that it is impossible to work together without the work taking a lot longer. There is the additional issue that, without a proper design process and documentation it actually MIGHT take longer.
One of the major advantages of coding being split from archive management, and all of the other overheads such as design and QA issues, is that there will be a managerial position who will have time to reform our coding process and maintain it. There will be a dedicated structure to set up training, and to support coders through various difficulties they might be having that is causing a delay or stress to the volunteer. There will be also a dedicated setup to follow what happened with individual tasks, to assign more people or mentors, and to ensure that coder related documentation (specs and APIs) are done and are up to date.
AO3 as a Product versus Experiment
If you have read through this whole thing so far you might at this point have either one of these questions.
- why do we need this extra setup when it has been working more or less
- why wasn’t this done the way it is from start.
Even when the open beta happened (which is right around when I joined the OTW) the archive was still an experiment. People tried it yes, and received accounts, but it was more of an indie project people fiddled around than used daily.
At some point through this changed. Today AO3 has over 60.000 registered users (and a lot more unregistered ones) some of which use this site as their primary fannish venue for activities such as posting work, running challenges, reccing, reading or commenting. At this point where people rely on uptime and continued quality of the website, the AO3 stops being an experiment and begins to be a product and needs to be treated as such.
What is the difference between a product an experiment in these definitions? Basically if you want to make a product you need to think about who you are marketing it to. There is a lot less flexibility for risky ideas, a lot less leeway to break things (no matter if you call it version 0.n or 1.n). There is the expectation to fix problems in reasonable speed, and to "not fail".
Being a product is something a lot of open source software fail at. Actually it is something even smaller companies fail at. It is the difference between being various linux distros, and ubuntu. The reason why companies like twitter, tumblr and facebook succeeded where many other startups fail.
This is important for AO3, because to succeed in its mission to be a home for fanwork to preserve it, it needs to be a hub. Right now most if not all of the alternatives of the AO3, major fandom archives, even those run by companies, are around the same caliber of product and service quality. Thus AO3 has no difficulty attracting users. However when next version of Fanlib comes around trying to monetize the fandom with no concern of their well being, will we be so lucky or will it come in form of a product of the polish level of pinterest or tumblr? How will then AO3 catch to be a viable alternative to the fans, since we do know average user do prefer usability over privacy and social concerns. It is for this reason if nothing else that AO3 needs aim for better.
If someone, a commercial someone comes in with a product, that might not care about fan interests but still offers a better product, people will migrate. Thus it is in AO3's best interest to be competitive.