Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

A stable Taverna 3 release is the target for this contract. (actual -  alpha 2 and buy in for the teams was achieved)

Timetable

Contract to start 23th of September 2013 to 20th of December 2013.

...

  • JIRA for tasks (all major work areas detailed in this document) - (from 02/09 meeting this needs to be re-emphasised) (20/12 - this was used well with 2 alpha rel)
  • David to send summary of work done on Friday 5pm (20/12 - started doing this - wiki pages and JIRA tasks - wiki pages references JIRA tasks - did this for 10 weeks - some taken over by testing and JIRA updated so auto pull out of tasks not working)
  • David to come in for face to face meetings on Mondays (usually) (02/09 - this was fine) (20/12 - done)

Things which will not be part of the Taverna 3 contract

...

  • Data Bundle specification/API
    • progress made in contract 1 (20/12 - developed a lot during use and then refined - driving provenance capture - not a version 1 yet)
    • need a release version (20/12 nearly at version 1)
    • Format for saving workflow runs (needs a discussion between David (run report requirements needed by saving step) and Stian) Taverna 2.x version ready by end of Sept 2013 (20/12 - done for Taverna 3 - report saving and added in)
  • Strategy for preservation of old CVS/SVN history when moving to GIT (02/09 - Donal to take the lead on this done by time we release software - can be done incrementally) (23/09 - David says more urgent because other are working on the code) (20/12 - not done for myGrid to do before going to GitHub)
    • restructure and move or
    • move and restructure
    • what are the pro's and cons
    • and who will do the actual move - David ? Donal ?
    • hard part: Agreed module structure by the team (02/09 Donal (leader), Stian, Alan (advice))
  • Provenance fit for purpose to capture information needed for SCAPE/BioVeL (02/09 - by end of Sept in good time for BioVeL plenary) (20/12 - we have provenance old system working - replaced with workflow report in data bundle - not in database now in data model - could turn existing provenance off - as could use data bundle - Stian needs to update prov exported to work from the data bundle so the old provenance system - new one not rely on old one uses monitor system rather than writing to db from the engine - in theory could replace engine - produced a speedup - still need a config for levels of provenance as still some overhead but not loads - new one more efficient and more extensible and fits in with model in the workbench)
    • save the provenance as well as the date from a run (David) (20/12/13 - yes everything is dated)
    • do we need a workflow run activity as part of Taverna 3 so you can query provenance, monitoring data and data outputs as part of a workflow (need a discussion) (02/09 - probably not needed by Dec release can come as a plugin later (Donal, Stian and Alan in 02/09 meeting)) (20/12 - not done)
    • re-engineering the provenance system to be extensible for each activity type - David to write up his ideas - this is 2-6 weeks work and has a lot of risk associated with it so perhaps not for Taverna 3.0.0 (02/09 - need to revisit this with David around - might need to do some sooner and 2-6 weeks to implement ? or document ?) (20/12 - and idea was that activities could put in monitorable properties e.g. wsdl activity - provenance capture at service side could send that to Taverna - by publish this - part of original design - normally how many things in the queue and how many jobs done - useful for use on a cluster - not tell one what did between being called and getting data)

Tasks

Source code layout (Donal) (20/12 - for myGrid to do)

  • Move to git - agreed Taverna 3 move to git - team to discuss level of history kept
  • Move to github - agreed

Forward port of Taverna 2.x changes since 2.4.0 to Taverna 3.x (new section 02/09) 

  • Discussion around Mark Borkum plugins (JSON, REST, OAuth) - myGrid to do work with support from David (02/09 Alan to own this area (maybe do the work post discussion)) (20/12 - Mark Borkum was pulled off of Taverna work)
  • Critical to get in components and interactions (blocking) - Will need David to work on components, interactions done by myGrid) (23/09 - it's actually Stian and Alan as test of plugin) (20/12 - for myGrid to do)
  • Dmitry WSDL parser - needs a discussion. (02/09 need to find out if it works with OSGi - Stian to own this one - needs a discussion when David is around) - put in a list of things to discuss when David's here to see if it is in scope - good task for learning the framework (not a blocker) - would update mechanism support this ? (20/12 - for myGrid to do)

Build

  • (02/09 - David to get the build working as soon as he starts so we can make it a baseline) - needs something runnable that jenkins build and for cmd tool can test as part of CI (20/12 - added some new modules for building products)
  • New module structure (draft - to be proposed by David W)
    • Core (engine)
    • Command Line (20/12 - done)
    • Workbench (20/12 - done)
    • Platform
    • Scufl2
    • Server (02/09 Donal - build structure should follow code structure) (20/12 - ask Donal)
  • Report on coding style guide usage as part of build process (02/09 - myGrid team does not feel this is critical path, maybe useful for Foundation - task to be done by David) (23/09 - SemanticVersioning check package that Stian found) (20/12 - PMD pluging for Maven will produce reports - need to run the report - question to myGrid when do you run it)
  • Copy and paste detection (02/09 - don't develop this by scratch - if there is something available then this can be used by David) (23/09 - pmd plugin from Maven) (23/09 - David said also not checking on impl  - it's just API's) (

...

  • 20/12 - PMD pluging for Maven will produce reports - need to run the report - question to myGrid when do you run it)

Plugin/Update system

A strong overall aim of this work is to enable a true continuous release mechanisms for Taverna. Most of the work will be hardening, documentation and porting (and any necessary bug fixes) as oppose to features. (23/09 - David confirmed this) (20/12 - David said this is what happened)

  • Documented process/routine for doing an update (20/12 - done)
    • How to do a quick patch? (“Service X needs new parameter Y!”, “Snow leoapard graphviz does not work!”) (20/12 - done apart from the graphviz case as can't be updated as not part of the build not run in Taverna call out to cmd to run dot)
    • How to do a bigger update (consistening of a set of updates) - current configuration -> new configuration : analyze to determine osgi bundle changes. (20/12 - done - included in part of docs how to combine the changes and use different channels)
    • say several APIs have changed in platform and we need new command line options and a new button in the Run dialogue (20/12 - covered)
  • How to do a new download?
    • How can it be picked up by old installations? Do they need to reinstall? (That should be OK as long as they are not left behind) - meeting Stian, Alan, David needed (20/12 build a new install - old install picks up what has changed - Taverna updated and Taverna downloaded should be identical installs - but do keep update history on old install - set of bundles is the same - need to decide frequency of builds and updates e.g. quarterly updates)
    • Effect on plugin developers? - meeting discussion and decide what to do - e.g. API need to be strictly extensive - can we live with this - discussion David and Donal - (related to impl dependencies)  (20/12 - not clear what this point is - but David documented the changes of what needed to be updated for new API) 
  • Plugins compatibility check against updates updates  (20/12 - auto done)
  • Plugin support (archetype, deployment) (20/12 - updated archetype and deployment plugin and added plugin for deploying a 3rd party plugin - when someone else gives us a plugin and hands that to us - maven plugin tool to install from files hard to do manually as checksummed)
  • Remove any remaining impl dependencies (may require new api/impl splits) (20/12 - still come as code structure - but many removed - one was to do with serialisation used xstream some entangled engine dependencies)
  • Porting of existing plugins (20/12 - for myGrid to do - did not port any)
    • Port myExperiment (as is) - Shoaib to talk to David about this - define the API/IMPL split - refer to workflows repository - maybe AstroTaverna with Stian - 23/09 - Stain to do AStroTaverna
  • Update architype for activity code For David (20/12 done by David)
    • Supported in tooling - Maven archetype to help make project plugins (20/12 - done)

NOTE - report need at 6pm on Friday - ask him to e-mail the Jira things he has been working on, note things he realised need to be worked that he is not working on .(20/12 - this was done in weekly reports and things were added to JIRA - David thinks everything was done)

Misc

  • Make modules more OSGi aware - services transient and not all there at the beginning - ACTION - ask David if this is done to a good enough level - 23/09 David says it's ongoing (20/12 - done)
  • Implement other service types
    • need to go through service types with Alan and see what needs to be done in context (e.g. tool service) - 23/09 check config with scufl2 (20/12 - all services done but not BioMoby - re-did all config - but not tool service because config not finalised - can run tool in alpha 1 but not in alpha 2 as that alpha2 had a updated config system - very complicated but only Alan really knows what it does - should use what is their and resist urge to refactor first)
  • Pickup config and preferences in one way (OSGi config service) - ask David if the way he has done it as a service - in which case no need to do the OSGi config - ask him how cmd line will deal with proxies (20/12 - tavern commons project used between products but itself not a product - not engine, not platform  but used by various services e.g. asking app what app home dir is - so all done)
  • Kymbat usability changes, fairly simple low hanging fruit ones - ACTION - Alan to go through with David to see what has been done and what has not (20/12 - done went through with Alex and annotated this - none of them are quick fixes)
  • improve iteration strategy on diagram - this is a must for c2 - get this working with the latest working of Dot (action in itself) - Taverna's dot parser - 2 weeks work 1 week test - sounds important - (donal used leopard version on mountain lion and it worked - not ideal) (20/12 - not done - for myGrid)
  • Scope out change needed for big data (e.g. reference service and the db backend) (20/12 not done - reference service might need a re-visit - might want to remove ref service and replace with the databundle (all a file or path - might be data or a reference to something) - change db backend to use databundle big API change then de-ref becomes trivial )
    • more of a documentation task - guidance of existing system as well as how to impove the system to handle bigger things
    • need a definiton (more than one disk, more than in memory, partitioning) - one of the team get David input - Donal to lead
  • show cardinalities on the diagram - 23/09 David to look into (20/12 - not done - discussed with Alan started looking at how might do this more for Taverna 2 - David showed Alan how to put things on the diagram)

People

Stain - 50% Taverna 3 and rest Provenance (20/12 - did databundle and saving wf report, testing - 50% me )
Alan - meetings and ? (20/12 meetings only and testing)
Alex N - 50% from mid- October (20/12 - just the last week 30% Tav 3)
Donal - 50% from October (20/12  - none)

AP (some testing)

SS (some testing)

Christian (1 week testing maybe a bit more)

Documentation (this looks like a month of work) documentation and fixing bugs

  • Coding style guide (20/12 - included - need to run reports - not done by David)
  • Testing standards guide (20/12 - myGrid to do - alpha2 testing was good)
  • Developers guide - (include osgi bundles, dependencies guide and versioning rules) (20/12 - done)
  • Plugin developers guide (20/12 done)
  • Plugin migration guide from 2.x to 3.x (20/12 done)

Testing

Integration (20/12 - look at testing for Alpha2)

Test top 20 workflows on myExperiment (none BioMoby ones)

...

Build workflows form the starter pack - Alex N

Unit testing (20/12 - another plugin in build that does this - but need to tell it to build the report)

Any new and modified code should be covered by appropriate unit test code coverage.

Report on test coverage to be provided.

Other - added 20/12

Please JIRA for other things that are not included in this document that were done or noted for Taverna 3 version alpha 1, alpha 2 and the proposal for Beta 1  (the latter being something myGrid should do)

MyGrid work (handover and post delivery)

...

  • add interaction service (this should be at the start of M5) - Alan to do - ideally in M7 - but could be after beta - depends on workload
  • port & refactor myExperiment plugin - after the beta
  • port & refactor BioCat plugin - after beta - Alan says needs a big re-design
  • Add component plugin (starting from existing 2.x Component plugin) - Alan and Stian to do - but after the beta - however - provenance is more important for SCAPE and BioVeL
  • write user guide to Taverna 3 - Alex N - do before beta - immediate pre-release
  • Update archetype for activity tutorials - Alex N - with the beta - or could come a month after (20/12 - done by David)
  • BioMart 0.8 in the Taverna 3 work - Donal - after beta
  • Verticals - Stian - take out defaults, look after branding, decide release or config - after beta (20/12 - service set configs are the same - should be easier to change than Tavern 2)
  •  
  • renderers discussion (keep or not) - early on in M5 (starting on 20/sept) - 23/09 - David to go through and test that they work - says lot's of copied code
  •  
  • Update the server to work with the new Taverna 3 command line - Donal
  • Testing on alternative exec environments e.g. running on a server - Donal - Shoaib's think with or soon after beta

...

Deliverables

  • Taverna 3 workbench (20/12 - alpha 2)
  • Taverna 3 platform service (OSGi) (20/12 - can be built as own product but not released)
  • Taverna 3 platform jar for none OSGi environments (20/12 - can be built as own product but not released)
  • Taverna 3 commandline (20/12 - alpha 2)
  • Taverna 3 server with alternative execution environment support (e.g. WB running a workflow on a server instance) (20/12 - not done - myGrid to do - stub to use Rob's Java interface to the Server and some work on Server side)
    • needs to update the server (use the updated command line in the first instance rather than the platform) - by (Donal)
  • Documentation (20/12 - please see section above)