Simple Black Box


in general:

  • messages/data i/o, multiple protocols
  • changes to filesystem, does logging
  • configuration that controls behavior

during development how do I test that >=1 daemon(s) behave correctly?

A test run...


set up sandbox (copy dir)


  • programatically modify configuration
    enable/disable settings
    set good/faulty params
  • verify initial state


  • set up probes (internally: tcpdump, ngrep, ..)
    set_http_probe swift "host $host_swift"
    set_http_probe tag "-d lo host lo and port $port"
  • launch process, capturing stdout,stderr,logs
  • send input to process, http request, upload file, ...


assert num_procs "$regex" "$num_procs" (pgrep)
assert listening "tcp:8080" (lsof)


kill_gracefull "$process"

remove_http_probe swift

remove_http_probe tag

assert num_procs "$regex" 0

assert listening "tcp:8080" 0


assert object_md5sum $swift_object

assert num_http_req swift 'GET /auth/v1.0' 1 1

assert http_resp swift "^PUT /v1/AUTH HTTP" 202

assert num_statsd_req lo 'upload.put:1|c' 1 1

assert no_errors $output/stderr_* $log (grep)

assert pattern "all is ok" $output/stdout

extra #1

use case:
  • events are scattered across M logfiles from app+probes
  • need central chronological overview of all events

extra #1

→ logstash session to ingest all events during SBB session → logstash as a local debugging tool

  • debug events for given testcase (distributed stacktrace)
  • compare behavior across test cases

extra #2 (WIP): profile-process

extra #2 (WIP): profile-process