Multicloud API bindings from documentation

Michał J. Gajda

Vitor Vitali Barrozzi

Gabriel Araujo

https://www.migamake.com

2020-04-04

Problem

  • Multiple cloud services
  • Ad-hoc REST APIs
  • Rapid changes need Infrastructure-as-a-Code (IaaC)
  • Tech stack inertia:
    • fixed SDK language
    • fixed container support
  • OpenAPI poorly supported by vendors (if at all)[SmartBear 2018]

Goals

  • Tech stack agility
    • any language
    • any library stack
    • fast replacement
  • Combine best available microservices
    • multi-vendor (AWS, GCP)
    • multi-platform (not just SDK)
    • no-code and low-code APIs

Solution

  • Code generation of API SDK bindings
  • Per-API scraper (Python, Scrapy[Evans])
  • Common format (Haskell)
  • Little generator code (Haskell)
  • Retargetable to other languages

Issues

  1. REST methods

  2. Call conventions

  3. Type conversion

Issues (1)

  1. REST methods:
  • GET
  • POST
  • PUT
  • DELETE
  • PATCH

Issues (2)

  1. REST methods

  2. Parameter passing conventions

HTTP request path variable:

POST /vpc/{vpcid}/status

URL-encoded query parameter:

?parameter=name

HTTP request header:

X-AMZ-Transport-encoding: chunked

Issues (2.1)

HTTP request path variable

URL-encoded query parameter

HTTP request header:

HTTP cookie:

Cookie: AUTH_TOKEN=8498904328099

HTTP auth header

HTTP request body

{'userid' : 819, name: 'Alice'}

Issues (3)

  1. REST methods

  2. Parameter passing conventions

  3. Type conversion

  • JSON
  • special strings: VPC id, userid, timestamp
  • examples only
  • use JSON Autotype

Dashboard

API ALL URL Parameters cURL Calls
backblaze 85% 100% 100% 85% 27
cloudflare 85% 100% 85% 97% 488
transferwise 91% 98% 96% 93% 81
gitlab 99% 99% 100% 99% 885
linode 13% 100% 13% 97% 196
azure 17% 99% 100% 17% 5461
site24x7 95% 99% 100% 96% 388
digitalocean 100% 100% 100% 100% 224
dropbox 81% 100% 100% 81% 234
mailgun 100% 100% 100% 100% 81

Implementation

Implementation

  1. Scraping

  2. Parsing and analysis

  3. Code generation

Implementation (1)

  1. Scraping[Evans]
  • Chrome driver
  • XPath[Clark 1999]
  • CSS[CSS Working Group 2011]
  • Regex, JQ, JSON Path

Implementation (2)

  1. Scraping[Evans]
  2. Parsing and analysis
  • API call path
  • JSON type inference
  • cURL scripts

Implementation (3)

  1. Scraping[Evans]
  2. Parsing and analysis
  3. Code generation:
  • Haskell monad stack
  • agile data format
  • retargetable to other languages [Michal J. Gajda 2020, Gajda]

Code

-- | From: 'http://documentation.mailgun.com/en/latest/api-domains.html'
type MailgunDomainsDELETE =
     (:>) "domains"
       ((:>) (Capture "domain" DomainType)
          ((:>) (BasicAuth "*" ())
             (Delete '[JSON] MailgunDomainsDELETEResponse)))
data MailgunDomainsDELETEArgs = MailgunDomainsDELETEArgs 
  { mailgunDomainsDELETEArgsBasicAuth :: BasicAuthType
  , mailgunDomainsDELETEArgsDomain    :: DomainType }
-- | Delete a domain from your account.
mailgunDomainsDELETE :: MailgunDomainsDELETEArgs
                     -> SC.ClientM MailgunDomainsDELETEResponse
mailgunDomainsDELETE args
  = SC.client (P.Proxy :: P.Proxy MailgunDomainsDELETE)
      (Just (mailgunDomainsDELETEArgsDomain args))
      (mailgunDomainsDELETEArgsBasicAuth args)

Benefits

  • thousands of API calls available within months;

  • easily retargetable;

  • easy scaling to other APIs;

  • No longer depends on vendor support;

  • Reduce supported code base

  • Available as paid service from https://migamake.com/

Question from audience

Limitations found in practice:

  • specialized components:
    • custom HTTP transport
    • custom auth
  • replicating bugs from documentation
  • some companies only seem to provide language-specific SDKs, no REST documentation

References

SmartBear (2018), “OpenAPI 3.0.2 Specification.” [Online]. Available: https://swagger.io/docs/specification/about/.

Evans S, “Scrapy.” [Online]. Available: https://scrapy.org/.

Clark J, (eds.) S D (1999), “XML path language (XPath) version 1.0,” W3C, W3C Recommendation.

CSS Working Group (2011), “Selectors Level 3.” [Online]. Available: https://www.w3.org/TR/2011/REC-css3-selectors-20110929/.

Michal J. Gajda D K (2020), “Fast XML/HTML tools for Haskell: XML Typelift and improved Xeno,” Manuscript under review.

Gajda M J, “Do not give us a bad name.”