My company is engaging with a multivariate testing vendor (I can't disclose which one just yet), and I am being asked to integrate their system into our flagship B2C commerce site.
Usually, the word "integrate" is a heavy-duty term. However, here it means to add a
<script> tag on several views, and then be out of the loop from that point forward. The mulitvariate experiments will be set up by our marketing department (and/or the vendor). Our development team will not be involved in that, and may not even be aware when it's happening.
id attributes, etc... and so if an experiment changes HTML elements too much, core commerce functionality will simply break altogether in unpredictable ways.
My concerns are amplified by the lack of a real test environment. The vendor does filtering, so that only AJAX calls from our production URL are processed. Calls from our Dev or QA environments are ignored. However, that's not the same thing as having a test environment. Rather, it means that production is now our test environment for each experiment.
Given all of the above, are there any practices available to mitigate some of these risks? Multivariate testing is such a new field, the first several pages of a Google search consists of shady consultants selling buzzwords to CIO's. There's not much written for the in-house technical audience ultimately responsible for keeping an eye on risk. Are there any ways to properly QA under the above restraints, or is there simply no alternative but to push back(*) in such a scenario?
(*) Note: Given the politics of the vendor relationship, we would need to build an extraordinarily airtight case for push-back, and it may be ignored anyway.