Jump to content

Deployment tooling/Cabal/2015-06-08

From mediawiki.org

June 8th

[edit]
  • conversation between Mukunda, Bryan: possibility of using git in scap deploy by buildling local repo from core + extensions
    • Middle-step git repository for mediawiki
    • binaries/blobs?
    • how do we handle submodules?
      • building local repo for deployment might ... ? (sorry, missed it)
    • Current services uses git via apache
      • Sometimes subjectively slow, may be room for improvement
    • Fan out
      • Preseeding, proxies needed -- could this be a post-merge jenkins job?
    • Deploy master node
      • Currently always tin
      • Could  start deploy from other proxies (pre-seed target)
      • Agnosticly built in terms of proxies vs pre-seed target
    • Might be good time to experiment with git based transport
      • Determine whether it's going to be over ssh/https/torrent/etc.
      • Run tests outside of prod to determine if feasible for MW
      • nginx/varnish, something, in each datacenter for fanout proxy
    • less clobbering
    • New work_dir after fetch with filecheckout

  *** Pros:   **** Scalable: github uses it   **** Supported: seems to be developed and supported by some of the core git team

    • Git Annex
    • GitTorrent
    • Git Fat
    • Criteria:

    *** Reliability (should check out the actual file not just text of sha)     *** Flexibility of transport mechanism     *** Scalability and reprudicibiliity (scales up and down)     *** Resource consuption      **** CPU     **** Limit network IO     **** Limit disk IO     *** Speed     *** Could possibly simulate cross-datacenter limitations with `tc`     *** eqiad - codfw RTT ~40ms, inside eqiad RTT is ~0.5ms

  • Definitions of success
    • Atomic failure modes
    • Optional fanout proxies based on git-http(s)
    • Blobs pused around with git annex or another solution (GitTorrent, large file store, figure out criteria for solution)
    • Continuous feedback to user
    • Verify integrity of repo
    • Not insanely slow (any slower than current system)

To use staging for deployment blob mechanisms