The PI noted that he has seen it to be helpful when more than one lab conducts an experiment, so that each can help the other avoid finicky and/or tacit experimental conditions that would prevent others from reproducing their results. I have wondered for some time (and for the case of microarrays in particular) whether the practice of “pair programming” that we use in software development would be more helpful than minimum information standards to increase the reproducibility of complex experiments. The problem with this, as the PI pointed out, is that duplicating every experiment can get expensive, and in the world of soft money (especially today’s world), people are always looking for ways to make the research dollar go farther. The possible long term efficiency of duplicating some efforts to increase data value and reduce a tendency to go down blind alleys might not be easy to quantify, and thus not easy to weigh quantitatively against the immediate penalty of “getting half as much work done”. (That’s certainly true in software.)

The PI pointed out that even if direct duplication was too expensive, he still advocated some kind of collaboration on experiments. In particular he advocated getting people together in the same room to look at the experiment together as it was being performed, so that the collaborator might catch important things that weren’t immediately apparent to the person performing the experiment. This, at most, only costs a small amount of travel funds.

I asked the PI if others shared his views, and he said that most of the larger microarray efforts had some sort of distributed work going on, but he wasn’t sure that this idea had been formalized anywhere.

I’m interested in this not only because of its parallel with software work, but also because I work for a company focused on facilitating collaborative science. I’m very interested in the different forms that scientific collaboration can take, and how best to help them along.

Rereading a paper months later often makes you wonder whether you read the paper at all the first time. This reading really clarified for me what the Z factor is, that it is not just for high-throughput screening, and raised a number of questions (especially after discussion with colleagues) not addressed in the paper.

The Z factor is the ratio of the “separation band” of the data to the assay dynamic range. A picture helps:

where μ₊ is the mean of the positive controls (in this case the controls with desired effect), μ_s is the mean of the data, σ₊ is the standard deviation of the positive controls, etc. The assay dynamic range in this diagram is μ₊ – μ_s. The screening window is then (μ₊ – μ_s) – (3σ₊ + 3σ_s), and the ratio of this to the dynamic range is the Z factor = 1 – (3σ₊ + 3σ_s)/(μ₊ – μ_s).

(If you’re reading this in an RSS reader, the story continues on my website.)

Zhang et al go on to describe desirable values of Z: Z = 1 is an “ideal assay” (the standard deviations are negligible compared to the difference between the means), 1 > Z ≥ 0.5 is an “excellent assay”, 0.5 > Z > 0 is a “double assay”, Z = 0 is a “yes/no assay” (no separation band, the two 3σ regions touch), and for Z < 0 “screening is essentially impossible”. Note that when the two distributions completely overlap (μ_s = μ₊), then Z is -∞.

As I mentioned above, the Z factor’s usefulness is not restricted to high-throughput screening assays. Indeed, it can be applied to any assay that measures a number of experimental subjects in an identical way and measures control values. However, when discussing application to assay optimization, the paper does point out that use of the Z factor requires “relatively large data sets”.

Three major questions arise:

1. What do these significant values of Z mean, especially 0.5? What is a “double” or “yes/no” assay?
2. How large is a “relatively large data set”, i.e. how large does your data set need to be to use the Z factor?
3. Why is the range of of Z -∞ to 1? Is there another parameter with a more intuitive range?

A nice companion reading for deeper understanding is the paper published in March 2007 by Sui and Wu. They take on question #1, by examining the statistical power of an assay at different Z factor values. The statistical power, in the case of a drug screening assay, is the probability that an active compound is scored as a hit (i.e. the probability of “true positives”). Z factor is calculated without referring to the hit threshold, the value beyond which a compound is scored as a hit. Often people score all outliers three standard deviations outside the mean as “hits” (in our diagram above, this would be any measurements falling above μ_s + 3σ_s, i.e. within the separation band or above). Sui and Wu show that if the standard deviations of sample data and controls are equal, then a Z factor of 0.5 corresponds to a statistical power of 0.999, i.e. there is only 0.1% chance that an active compound is not scored as a hit.

However, they also show that if the standard deviations are not equal, then interpreting the Z factor becomes considerably trickier. They also show that although the Z factor calculation does not necessarily rely on the error distributions being normal, for non-normal error distributions (where the sample (or the control data) is not well described by the normal distribution N(μ_s, σ_s²)) the Z factor does a poor job of describing the reliability of the assay, demonstrated by the fact that the Z factor is different if non-normally distributed data is transformed to be closer to normal.

Sui and Wu suggest caution when interpreting Z (and Z’) factor values, and recommend that analysts confirm normality of the data (transforming if necessary) and calculate the statistical power corresponding to the distributions of the sample/control data and the hit threshold to get a more reliable measure of assay reliability.

Humorous side note: Sui and Wu interpret “double assay” to mean “doable assay”. Who knows.

As for question #2 (“how large does a data set need to be to get a reliable estimate of Z factor?”), this can be calculated from the standard error of the estimators used to calculate the means and standard deviations. That is, when calculating Z factor, one can’t actually use the real means and standard deviations of the underlying distributions of the samples and controls, one can only estimate these quantities by making a number of measurements of the samples and controls. I plan to calculate these and publish them here at some point, but any statistician can do the same (probably more efficiently and correctly than I).

Finally, question #3: Zhang et al‘s objective was to develop a dimensionless constant that took three distributional parameters into account: the difference of the means of the samples and controls, the variability of the controls and the variability of the sample data. There are other ways to combine these parameters into dimensionless constants that are different from the Z factor. For instance, one could calculate the ratio of the separation band to the sum of (3σ₊ + 3σ_s), call this parameter C. C varies between -∞ and +∞, and if σ is the same for controls and samples, then C = -1 when Z = -∞, C = 0 when Z = 0, C = 1 when Z = 0.5, and C = +∞ when Z = 1.

This gets rid of the weird “0.5″ and the upper limit of 1, but I think that actually the Z factor is a more accurate reflection of reliability. The reason is that the reliability of an assay with Z of 0.9 vs. 1.0 is really quite small, even if the dynamic range in these two cases is very different, because 0.9 is already good enough. By contrast, the difference between the reliability of an assay with Z = 0 and Z = -∞ is huge, because we go from having the data variability ranges ([μ - 3σ, μ + 3σ] with appropriate subscripts) touch to having the two distributions completely overlap. By contrast C would only vary from 0 and -1 between these two cases.

I may retouch the explanation above for clarity at some point, especially if people ask questions.

1. location independence — an LSID identifies a resource, not a particular data record on a particular server on the network
2. global uniqueness — the same LSID should never be used to refer to two distinct objects
3. local assignment — there must be some easy mechanism for data object creators to assign globally unique LSIDs to their data without the need for onerous bureaucracy
4. permanence — once assigned, an LSID cannot be reassigned to a different object
5. semantic opacity — data clients who use LSIDs to refer to data are not supposed to read into the substructure of the LSID and make any conclusions about what the LSID means, other than being a key for a particular data object, e.g. urn:lsid:frank:dog:38922 should not be assumed to have anything to do with Frank or his dog.
6. data and metadata — in addition to the bytes representing the data object identified, there is a separate “channel” of information, the metadata, which can contain information about relationships between the given data object and other data objects.

and…, some other stuff, see the official spec. LSIDs look like this: urn:lsid:ncbi.nlm.nih.gov:pubmed:9486653. One of the key steps in employing LSIDs on a grid (or in any network accessible way) is the deployment of some kind of resolution service, a server or set of servers that delivers the data for a given LSID to a requesting data client. Also, as a corollary of the “global uniqueness” requirement above, it is important that requesting resolution of a given LSID should always return the same set of bytes, regardless of when the resolution occurs. That is, if you assign an LSID to an object, you assign it to a specific set of bytes representing that object, and that association can never change (although the data provider might stop providing the data). This allows you to cache these objects without worrying about cache invalidation (since the bytes will never change) and more importantly enables reproducible research: a computational researcher can publish results based on a dataset identified by a bunch of LSIDs, and another researcher later can run the same computations on those same LSIDs and expect to arrive at exactly the same results.

Importantly, the metadata referred to by an LSID is not required to be byte-identical, so it has a bit more flexibility to cover some important use cases, as we will see in my next post on the subject.

They recently posted screenshots (and, more recently, these) of the product under development, and you can sign up to get in on internal beta testing at scilnk.com.

Full disclosure: I was personally working with the SciLnk team in its early days, but quickly realized I had bitten off more than I could chew, what with having a full-time day job and not living in Boston with the rest of the crew. I no longer have any financial interest in the company, I just think it’s a cool idea.

In my last post I argued for the central importance of effective collaboration and communication in biomedical informatics. I wanted to list a few things that have worked for my teams in those areas. At Northwestern we worked on two projects. Neuromice.org is a phenotype database and virtual storefront for the mutant lines produced by three neurologically-focused whole genome mutagenesis efforts at Northwestern, the Jackson Laboratory and the Tennessee Mouse Genome Consortium. The other application, MouseDB, is an intranet (i.e. you can’t see it) colony and phenotyping management system for the mice under study at Northwestern (10,000 mice/year when we were in full swing). Each project had different challenges, but here are a few things I learned from those experiences. Some are pretty standard agile ideas, others less so.

• Each distinct customer/user subgroup should appoint a representative who speaks for that subgroup in all discussions of feature definition and priority. Keep the number of subgroups as small as possible (ideally, one). This greatly reduces the uncertainty and difficulty of scope decisions.
• Some users in the group might have no reason to use your software. Make this fact explicit, and don’t factor their interests into the product.
• Be completely open with your user community. Give them the opportunity to know everything you’re working on, and the reasons for (and the opportunity to contribute to) any decisions made about features going into the product.
• A development team should avoid making any decisions about scope or feature priority. Emphasize to users that it is in their power to steer the software toward the greatest possible utility. Technical improvements are a sticking point here, but we’ve found if you make a good argument for them, users understand their value and will prioritize them appropriately.
• If you let academics’ busy schedules eat away at your face time with them, you will eventually suffer for it. Be creative.

I think I’ve reached my word limit. Over the next couple weeks I’m going to let loose a flurry of technical posts on various topics that have been on my mind lately.

There are certainly success stories. Important approaches to biological analysis in use today were not available ten years ago. I think some of the frustration arises because, in spite of these successes, some users feel that much of the biomedical software being churned out today just isn’t quite useful enough to justify the cost (in money and time) of using it. Assuming this is the case, we must then ask, why? Is biological analysis too difficult to capture in a set of machine instructions? Are bioinformaticians just a bunch of good for naughts?

The latter response, though intended to be humorous, is actually probably more common among biologists than we bioinformaticians might like. This answer also suggests a dichotomy too often encountered in organizations undertaking software development, that of users vs. developers, or, in the domain-specific vernacular, scientists/clinicians vs. bioinformaticians. Users, angered by the failure of software, blame developers for not working hard enough, for not listening, for being idiots (you know they think this sometimes), etc. Developers, for their part, are no more charitable. Developers blame users for not knowing what they want, for using software inconsistently, for not being able to work around seemingly trivial problems, and of course for being idiots. Much of the naturally occurring tension churned up in the process of building software finds its release in similar fits of whinging by one camp or the other. When we’re more reasonable, we are still honestly perplexed by the question, why isn’t this working out better?

In the past I’ve heard people say that bioinformaticians just need to be trained very well in both biology and computer science, that this would alleviate a lot of the problem of getting them to build biologically relevant and valuable software. This may work in some cases. A couple weeks ago I was having lunch with a biologist colleague, and he told me that I needed to learn the biology better, otherwise I would always be beholden to biologists to come up with interesting problems to work on. I see what he was getting at, but I don’t think that is the solution. The truth is both biology and software development are so complex that I don’t think it’s possible to gather into one person’s head all the expertise necessary to produce all the products that bioinformatics promises. Rather, I think the answer is better communication between biology experts and software experts.

Rather than focusing solely on algorithms and technologies, we must focus more on the people side of building biomedical software. You read this very comment in the bio-IT business literature sometimes, taken from the mouths of venture capitalists, in the form of something like “companies can no longer expect to get funding simply for having cool technology”, their software has to solve a biologically relevant problem, i.e. it has to be useful. I am reminded of something I heard during a talk at an agile conference in New Orleans in 2003. Josh Kerievsky said, and I paraphrase, “Some think we’re in the technology business, but we’re not. We’re in the communication business”. Communicating effectively with users is surprisingly difficult to do, and requires wisdom and dedication to get right. Effective communication is a much bigger challenge than the algorithms and the technology usually are. What’s more, it’s a two-way street, and both developers and scientists have to be committed.

I think biologists and bioinformaticians want to communicate better. I think part of the problem is organizational. For instance, at Northwestern, like at any university, it’s very difficult to get good office space. We were stuck in a converted greenhouse for the three years I was in Evanston, on the top floor of the Hogan building. At first we were in the same building as many of the biologists with whom we worked, but not all of them. Over the course of the project, a nice new building opened up, and a number of them moved into the new building. We typically saw these people once a week, if that, at weekly user meetings. We tried to keep contact with them by going to see them individually on a periodic basis (although I think we probably could have done a better job at that). But these people are very busy, and it’s difficult to fit into their schedules. As my biologist colleague pointed out at lunch, it would have been ideal if we could have had shared office space, so that spontaneous discussions would have been more frequent. I think the software we built would have become more useful as a result.

There are many other organizational problems (the difficulty of funding ongoing bioinformatics groups on grants; the lack of a history of operational management positions within academic groups). I imagine some of these get easier in industry. But these are not the only problems. I also think that we don’t yet focus enough attention on communication, on doing it right, and on getting help from outside to do it right. Most of us assume that, hey, we’re smart people, we should be able to communicate. Part of the problem is we’re too smart. We’re trying to communicate complex information, information we’re used to communicating with peers within our field who usually understand us even if we’re not clear. The level of tacit knowledge in biology, like in software, is very high, and it often takes people outside the field quite a while to get a feel for a problem.

This post is long enough. We talk about some of these communication issues in the paper David Kane and others and I are trying to get published on agile software methods in bioinformatics. I think agile principles help, at the very least because they get software engineers to focus less on impressing people with their bulletproof processes and more on people and on communication that works (i.e. “Individuals and interactions over processes and tools”). Although our application domain is more technical, the general software industry and its customers have been grappling with these issues for years, and there are a number of very smart and capable people out there who could really help transfect bioinformatics groups with good approaches for tackling these problems.

This idea is still very new, but I will keep you posted on developments as things unfold.

• Establish personal contacts, bringing together those responsible for research computing activities at biomedical research institutions
• Identify and document common problems and interests
• Seek opportunities for partnership / consortium activities
• Identify common issues that should be brought to attention of home institutions, government and other funding agencies

One of the things I really liked about BRIITE was its focus on stimulating offline discussions. I went to the meeting hoping to find out if there were others in the biomedical informatics community who cared about software development issues. The focus of the meeting was “IT Support for Multi-Institution Collaborative Research”. We heard talks on federated identity management, Globus, GridShib, BIRN, and, slightly off-topic but still pretty neat, the Research Channel. So, the meeting was mostly about distributed security, and many attending were not primarily software development people.

Due to the focus on offline discussions, attendees are encouraged to suggest topics of interest, and then others sign up to discuss these topics as a group (a conference practice I’ve heard called a “Birds of a Feather” session). I proposed software development practices as a topic, and and a small group of us got together for a lively two-hour discussion. I will summarize what we talked about here.

People were interested in the topic for a variety of reasons. One person wanted to know how to introduce better development practices to his group. Another wanted input on how to manage many small projects at once (a common problem in biomedical informatics). Another was plagued with the problem of funding shared informatics resources at a biomedical research institution (people clearly need these resources, but no one wants to or can pay for them). Another wanted to learn more about agile software methods. Finally, I wanted to talk about how better to promote awareness and dicussion of software practices in the biomedical informatics community.

In the interest of brevity I’ll just list bullet points rather than go on and on about each item.

Funding IT

While not strictly a software development practice topic, anyone who wants to assemble a decent-sized software team at a biomedical research institution runs up against the problem of funding it. Most of the helpful tips here came from Charles Donnelly of the Jackson Laboratory.

• If these resources don’t exist, who provides seed funding to get them started? Investigators typically cannot fund these things from their research grants. Institutional funding and commitment is necessary. Where it exists, people have seen some success in developing shared informatics resources (e.g. the Jackson Laboratory).
• Funding improves if core informatics staff reviews research grants to help make them more reaslitic about informatics needs and funding. Encouraging awareness among scientific investigators and administrators is key.
• Once you have institutional funding, keep track of the time you spend on various grants. Calculate the percentage of your total work this amounts to, and tell the PIs on those grants. This will help them understand how much their informatics costs, even if they aren’t paying for it yet.
• Generally, PIs that have done some computing in the past are more forward-looking, so these are good people to start with (no-brainer).
• Funding informatics from research grants is difficult, because all budgeting is done ahead of time. Often, however, the informatics needs change over the course of the scientific project. How do you come up with specifics like $ and number of FTEs before a project begins? (The answer, it seems to me, is you do what scientists do. You make your best guess, and then you adjust what you produce based on what you have the money to produce once you learn what is possible during the course of the project.)

Project Management

Mix of suggestions and problems here . . .

• One way to scale your capabilities effectively is to develop shared resources used by several groups. Supporting shared resources can be a challenge, however, because PIs may want results formatted their own way, not in a common format.
• Before embarking on a project, draw up a project charter. This document establishes the level of involvement of key people (on both the informatics and science side) and the high-level outline of the work planned. The process of preparing this document will let you know if science people are invested enough in the project, or if there are reasons to worry. One cannot underestimate the importance of buy-in and personal investment in the success of an informatics endeavor by the science leaders on the project.
• Promote awareness of software development: Jax has a software lifecycle process approved by the institution and posted up everywhere.

Requirements/Communication

• Some used a detailed requirements gathering process (at Jax they draw up 150 page documents) to firm up what would be developed.
• Others used agile approaches, which are interactive with the scientists throughout the life of the project and make it a specific point to allow the requirements to evolve along with the scientists’ understanding of their needs.
• All agreed that requirements gathering in biomedical software has to be an ongoing, interactive endeavor – biologists sometimes want to wait until features are in production before they give feedback, they want to see something working and tinker with it.
• Underestimated opportunity for software people: helping biologists with their ad-hoc solutions (Access, Excel, FileMaker Pro) — biologists will always use Excel, so is there some way we can help them use Excel better? Can we leverage their familiarity with these technologies? A software development background teaches you to look askance on “low-tech” solutions like these, but in this domain there is something to be said for them.
• Starting with the front end (UI) and working back helps user interaction — you can show a prototype of an interface and get better feedback earlier.
• Best ideas don’t always come from biologists: we are doing our job best when we are helping scientists understand what is possible with software, and helping them focus on their goals and how software could help achieve them.

Looking Forward

We talked about possible content for a meeting or conference on biomedical and bioinformatics software development. People showed interest in the following.

• Documenting typical staffing models — help us all understand our options for organizing our work and getting it funded
• Questionnaire — find out what people are doing, try to get a picture of the current state of affairs
• Present experience reports: What working examples are out there?
• People preferred a workshop more than a conference, a hands-on experience. Plenty of short talks that don’t give answers, but raise questions – followed by open discussions.
• People would like to see defined milestones for the meeting.
• Tutorials on some subjects (e.g. project management, quality assurance and testing, good development practices) would be welcome
• Q&A sessions – what have you done that can benefit others, what are you doing?